This document contains ais object commands - the commands to read (GET), write (PUT), APPEND, PROMOTE, PREFETCH, EVICT etc. user data.

Namely:

1 $ ais object <TAB-TAB>
2 
3 get     put        cp           etl          set-custom   prefetch     show      cat
4 ls      promote    archive      concat       rm           evict        mv

GET object

Use ais object get or, same, ais get to GET data from aistore. In other words, read data from the cluster and, optionally, save it locally.

ais get BUCKET[/OBJECT_NAME] [OUT_FILE|-] [command options]

there’s

a bucket source with an optional object name (BUCKET[/OBJECT_NAME]), and
destination (but also optional) [OUT_FILE] or standard output (-)

Here’s in detail:

1 $ ais get --help
2 
3 NAME:
4    ais get - (alias for "object get") Get an object, a shard, an archived file, or a range of bytes from all of the above;
5               write the content locally with destination options including: filename, directory, STDOUT ('-'), or '/dev/null' (discard);
6               assorted options further include:
7               - '--prefix' to get multiple objects in one shot (empty prefix for the entire bucket);
8               - '--extract' or '--archpath' to extract archived content;
9               - '--progress' and '--refresh' to watch progress bar;
10               - '-v' to produce verbose output when getting multiple objects.
11 
12 USAGE:
13    ais get BUCKET[/OBJECT_NAME] [OUT_FILE|OUT_DIR|-] [command options]
14 
15 OPTIONS:
16    --archive            List archived content (see docs/archive.md for details)
17    --archmime value     Expected format (mime type) of an object ("shard") formatted as: .tar, .tgz or .tar.gz, .zip, .tar.lz4;
18                         especially usable for shards with non-standard extensions
19    --archmode value     Enumerated "matching mode" that tells aistore how to handle '--archregx', one of:
20                           * regexp - general purpose regular expression;
21                           * prefix - matching filename starts with;
22                           * suffix - matching filename ends with;
23                           * substr - matching filename contains;
24                           * wdskey - WebDataset key
25                         example:
26                           given a shard containing (subdir/aaa.jpg, subdir/aaa.json, subdir/bbb.jpg, subdir/bbb.json, ...)
27                           and wdskey=subdir/aaa, aistore will match and return (subdir/aaa.jpg, subdir/aaa.json)
28    --archpath value     Extract the specified file from an object ("shard") formatted as: .tar, .tgz or .tar.gz, .zip, .tar.lz4;
29                         see also: '--archregx'
30    --archregx value     Specifies prefix, suffix, substring, WebDataset key, _or_ a general-purpose regular expression
31                         to select possibly multiple matching archived files from a given shard;
32                         is used in combination with '--archmode' ("matching mode") option
33    --blob-download      Use blob-downloader to fetch large objects from remote backend into AIStore cluster (see docs/blob_downloader.md)
34    --mpd                Use multipart download to read large objects from AIStore cluster to the client-side;
35                         for single-object only; use '--chunk-size' and '--num-workers' to configure
36    --cached             Only get in-cluster objects, i.e., objects from the respective remote bucket that are present ("cached") in the cluster
37    --check-cached       Check whether a given named object is present in cluster
38                         (applies only to buckets with remote backend)
39    --checksum           Validate checksum
40    --chunk-size value   Chunk size in IEC or SI units, or "raw" bytes (e.g.: 4mb, 1MiB, 1048576, 128k; see '--units')
41    --extract, -x        Extract all files from archive(s)
42    --inv-id value       Bucket inventory ID (optional; by default, we use bucket name as the bucket's inventory ID)
43    --inv-name value     Bucket inventory name (optional; system default name is '.inventory')
44    --inventory          List objects using _bucket inventory_ (docs/s3compat.md); requires s3:// backend; will provide significant performance
45                         boost when used with very large s3 buckets; e.g. usage:
46                           1) 'ais ls s3://abc --inventory'
47                           2) 'ais ls s3://abc --inventory --paged --prefix=subdir/'
48                         (see also: docs/s3compat.md)
49    --latest             Check in-cluster metadata and, possibly, GET, download, prefetch, or otherwise copy the latest object version
50                         from the associated remote bucket;
51                         the option provides operation-level control over object versioning (and version synchronization)
52                         without the need to change the corresponding bucket configuration: 'versioning.validate_warm_get';
53                         see also:
54                           - 'ais show bucket BUCKET versioning'
55                           - 'ais bucket props set BUCKET versioning'
56                           - 'ais ls --check-versions'
57                         supported commands include:
58                           - 'ais cp', 'ais prefetch', 'ais get'
59    --length value       Object read length; default formatting: IEC (use '--units' to override)
60    --limit value        The maximum number of objects to list, get, or otherwise handle (0 - unlimited; see also '--max-pages'),
61                         e.g.:
62                         - 'ais ls gs://abc/dir --limit 1234 --cached --props size,custom,atime'  - list no more than 1234 objects
63                         - 'ais get gs://abc /dev/null --prefix dir --limit 1234'                 - get --/--
64                         - 'ais scrub gs://abc/dir --limit 1234'                                  - scrub --/-- (default: 0)
65    --num-workers value  Number of concurrent workers for --blob-download or --mpd; system default when omitted or zero (default: 0)
66    --offset value       Object read offset; must be used together with '--length'; default formatting: IEC (use '--units' to override)
67    --prefix value       Get objects with names starting with the specified prefix, e.g.:
68                         '--prefix a/b/c' - get objects from the virtual directory a/b/c and objects from the virtual directory
69                         a/b that have their names (relative to this directory) starting with 'c';
70                         '--prefix ""' - get entire bucket (all objects)
71    --progress           Multi-object progress: show progress bar for number of objects processed (see 'GET multiple objects' below)
72    --refresh value      Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
73                         valid time units: ns, us (or µs), ms, s (default), m, h
74    --silent             Server-side flag, an indication for aistore _not_ to log assorted errors (e.g., HEAD(object) failures)
75    --skip-lookup        Do not execute HEAD(bucket) request to lookup remote bucket and its properties; possible usage scenarios include:
76                          1) adding remote bucket to aistore without first checking the bucket's accessibility
77                             (e.g., to configure the bucket's aistore properties with alternative security profile and/or endpoint)
78                          2) listing public-access Cloud buckets where certain operations (e.g., 'HEAD(bucket)') may be disallowed
79    --units value        Show statistics and/or parse command-line specified sizes using one of the following units of measurement:
80                         iec - IEC format, e.g.: KiB, MiB, GiB (default)
81                         si  - SI (metric) format, e.g.: KB, MB, GB
82                         raw - do not convert to (or from) human-readable format
83    --verbose, -v        Verbose output
84    --yes, -y            Assume 'yes' to all questions
85    --help, -h           Show help

Save object to local file

Get the imagenet_train-000010.tgz object from the imagenet bucket and write it to a local file, ~/train-10.tgz:

1 $ ais get ais://imagenet/imagenet_train-000010.tgz ~/train-10.tgz
2 GET "imagenet_train-000010.tgz" from bucket "imagenet" as "/home/user/train-10.tgz" [946.8MiB]

For comparison, the same GET using curl and the two supported variants of RESTful API:

1 # 1. curl GET using conventional RESTful API
2 # (`aistore` in the URL is a host that runs any AIStore gateway that can be specified via `AIS_ENDPOINT` environment):
3 
4 $ curl -L -X GET 'http://aistore/v1/objects/imagenet/magenet_train-000010.tgz?provider=gs -o ~/train-10.tgz'
5 
6 # 2. and the same using "easy URL":
7 
8 $ curl -L -X GET 'http://aistore/ais/imagenet/magenet_train-000010.tgz -o ~/train-10.tgz'

Save object to local file with implied file name

If OUT_FILE is omitted, the local file name is implied from the object name.

Get the imagenet_train-000010.tgz object from the imagenet bucket and write it to a local file, imagenet_train-000010.tgz:

1 $ ais get imagenet/imagenet_train-000010.tgz
2 GET "imagenet_train-000010.tgz" from bucket "imagenet" as "imagenet_train-000010.tgz" [946.8MiB]

Get object and print it to standard output

Get the imagenet_train-000010.tgz object from the imagenet AWS bucket and write it to standard output:

1 $ ais get aws://imagenet/imagenet_train-000010.tgz -

Check if object is cached

We say that “an object is cached” to indicate two separate things:

The object was originally downloaded from a remote (e.g., 3rd party Cloud) bucket, a bucket in a remote AIS cluster, or a HTTP(s) based dataset;
The object is stored in the AIS cluster.

In other words, the term “cached” is simply a shortcut to indicate the object’s immediate availability without the need to go to the object’s original location. Being “cached” does not have any implications on an object’s persistence: “cached” objects, similar to objects that originated in a given AIS cluster, are stored with arbitrary (per bucket configurable) levels of redundancy, etc. In short, the same storage policies apply to “cached” and “non-cached”.

The following example checks whether imagenet_train-000010.tgz is “cached” in the bucket imagenet:

1 $ ais get --cached ais://imagenet/imagenet_train-000010.tgz
2 Cached: true

Read range

Get the contents of object list.txt from texts bucket starting from offset 1024 length 1024 and save it as ~/list.txt file:

1 $ ais get --offset 1024 --length 1024 ais://texts/list.txt ~/list.txt
2 Read 1.00KiB (1024 B)

Example: read-range multiple objects

Let’s say, bucket ais://src contains 4 copies of aistore readme in its virtual directory docs/:

The following reads 10 bytes from each copy and prints the result:

1 $ ais get ais://src --prefix "docs/" --offset 0 --length 10 -
2 Read range 4 objects from ais://src to standard output (total size 50.23KiB) [Y/N]: y
3 
4 **AIStore **AIStore **AIStore **AIStore $

Same as above with automatic confirmation and writing results to /tmp/w:

1 $ ais get ais://src --prefix "docs/" --offset 0 --length 10 /tmp/w -y
2 
3 $ ls -al /tmp/w | awk '{print $5,$9}'
4 
5 10 README.md
6 10 copy1.md
7 10 copy2.md
8 10 copy3.md

Multipart Download

Use --mpd for client-side concurrent range-based download with built-in progress bar. This is useful for large objects where you want to see download progress and potentially benefit from parallel chunk downloads.

--mpd vs --blob-download: These are different mechanisms for different purposes:

--mpd (multipart download): Client-side - downloads object from AIStore to client using concurrent range requests

--blob-download: Server-side - fetches object from remote backend (e.g., S3, GCS) into AIStore cluster for caching; see blob_downloader.md

Note: --mpd is for single-object download only and includes its own progress bar automatically. For multi-object downloads (using --prefix), use --progress instead to track the number of objects processed.

1 # Basic multipart download with progress bar
2 $ ais get ais://bucket/large-file.bin ./local-file.bin --mpd
3 large-file.bin 4.00 GiB / 4.00 GiB [======================================] 00:00:15 1.2 GiB/s
4 GET ais://bucket/large-file.bin
5 
6 # With custom chunk size and number of workers
7 $ ais get s3://bucket/huge-object ./output --mpd --chunk-size 16mb --num-workers 32
8 large-file.bin 4.00 GiB / 4.00 GiB [======================================] 00:00:15 1.2 GiB/s
9 GET s3://bucket/huge-object

GET multiple objects

Use --prefix to download multiple objects at once. Note that destination in this case is a local directory and that (an empty) prefix indicates getting entire bucket; see --help for details.

Use --progress to show multi-object progress (number of objects processed). This is different from --mpd which has its own built-in progress bar for single-object downloads.

1 $ ais get s3://abc /tmp/w --prefix "" --progress
2 GET 60 objects from s3://abc to /tmp/w (size 92.47MiB) [Y/N]: y
3 Objects:                     59/60 [============================================================>-] 98 %
4 Total size:  63.00 MiB / 92.47 MiB [=========================================>--------------------] 68 %

GET archived content

For objects formatted as (.tar, .tar.gz, .tar.lz4, or .zip), it is possible to GET and extract them in one shot. There are two “responsible” options:

Name	Description
`--archpath`	extract the specified file from an archive (shard)
`--extract`	extract all files from archive(s)

Maybe the most basic:

Example: extracting one file using its fully-qualified name::

1 $ ais get ais://nnn/A.tar/tutorials/README.md /tmp/out --archive

assuming, ais://nnn/A.tar was previously created via (e.g.) ais archive put docs ais://nnn/A.tar -r

Example: extract all files from all shards with a given prefix

Let’s say, there’s a bucket ais://dst with a virtual directory abc/ that in turn contains:

1 $ ais ls ais://dst --prefix abc/
2 NAME             SIZE
3 abc/A.tar.gz         5.18KiB
4 abc/B.tar.lz4        247.88KiB
5 abc/C.tar.zip        4.15KiB
6 abc/D.tar            2.00KiB

Next, we GET and extract them all in the respective sub-directories (note also the --verbose option):

1 $ ais get ais://dst /tmp/w --prefix "abc/" --extract -v
2 
3 GET 4 objects from ais://dst to /tmp/w (total size 259.21KiB) [Y/N]: y
4 GET D.tar from ais://dst as "/tmp/w/D.tar" (2.00KiB) and extract as /tmp/w/D
5 GET A.tar.gz from ais://dst as "/tmp/w/A.tar.gz" (5.18KiB) and extract as /tmp/w/A
6 GET C.tar.zip from ais://dst as "/tmp/w/C.tar.zip" (4.15KiB) and extract as /tmp/w/C
7 GET B.tar.lz4 from ais://dst as "/tmp/w/B.tar.lz4" (247.88KiB) and extract as /tmp/w/B

Example: use ‘—prefix’ that crosses shard boundary

For starters, we recursively archive all aistore docs:

1 $ ais put docs ais://A.tar --archive -r

To list a virtual subdirectory inside this newly created shard (e.g.):

1 $ ais archive ls ais://nnn --prefix A.tar/tutorials
2 NAME                                             SIZE
3     A.tar/tutorials/README.md                    561B
4     A.tar/tutorials/etl/compute_md5.md           8.28KiB
5     A.tar/tutorials/etl/etl_imagenet_pytorch.md  4.16KiB
6     A.tar/tutorials/etl/etl_webdataset.md        3.97KiB
7 Listed: 4 names

Now, extract matching files from the bucket to /tmp/out:

1 $ ais get ais://nnn --prefix A.tar/tutorials /tmp/out --archive
2 GET 6 objects from ais://nnn/tmp/out (total size 17.81MiB) [Y/N]: y
3 
4 $ ls -al /tmp/out/tutorials/
5 total 20
6 drwxr-x--- 4 root root 4096 May 13 20:05 ./
7 drwxr-xr-x 3 root root 4096 May 13 20:05 ../
8 drwxr-x--- 2 root root 4096 May 13 20:05 etl/
9 -rw-r--r-- 1 root root  561 May 13 20:05 README.md
10 drwxr-x--- 2 root root 4096 May 13 20:05 various/

The result:

1 $ tree /tmp/out
2 /tmp/out
3 ├── A.tar
4 └── tutorials
5     ├── etl
6     │   ├── compute_md5.md
7     │   ├── etl_imagenet_pytorch.md
8     │   └── etl_webdataset.md
9     ├── README.md
10     └── etl
11         └── compute_md5.md
12         └── etl_imagenet_pytorch.md
13         └── etl_webdataset.md

NOTE: for more “archival” options and examples, please see docs/cli/archive.md.

Print object content

ais object cat BUCKET/OBJECT_NAME

Get OBJECT_NAME from bucket BUCKET and print it to standard output. Alias for ais get BUCKET/OBJECT_NAME -.

Options

1 $ ais object cat --help
2 
3 NAME:
4    ais object cat - Print object's content to STDOUT (same as Linux shell 'cat')
5 
6 USAGE:
7    ais object cat BUCKET/OBJECT_NAME [command options]
8 
9 OPTIONS:
10    --archpath value  Extract the specified file from an object ("shard") formatted as: .tar, .tgz or .tar.gz, .zip, .tar.lz4;
11                      see also: '--archregx'
12    --checksum        Validate checksum
13    --force, -f       Force execution of the command (caution: advanced usage only)
14    --length value    Object read length; default formatting: IEC (use '--units' to override)
15    --offset value    Object read offset; must be used together with '--length'; default formatting: IEC (use '--units' to override)
16    --help, -h        Show help

Print content of object

Print content of list.txt from local bucket texts to the standard output:

1 $ ais object cat ais://texts/list.txt

Read range

Print content of object list.txt starting from offset 1024 length 1024 to the standard output:

1 $ ais object cat ais://texts/list.txt --offset 1024 --length 1024

Show object properties

ais object show [--props PROP_LIST] BUCKET/OBJECT_NAME

Get object detailed information. PROP_LIST is a comma-separated list of properties to display. If PROP_LIST is omitted, default properties are shown.

Supported properties:

cached - the object cached on local drives (always true for AIS buckets)
size - object size
version - object version (empty if versioning is disabled for the bucket)
atime - object’s last access time
copies - the number of object replicas per target (1 if bucket mirroring is disabled), and mountpath where object and its mirrors are located
checksum - object’s checksum
node - on which target the object is located
ec - object’s EC info (empty if EC is disabled for the bucket, if EC is enabled it looks like DATA:PARITY[MODE], where DATA - the number of data slices, PARITY - the number of parity slices, and MODE is protection mode selected for the object: replicated - object has PARITY replicas on other targets, encoded the object is erasure coded and other targets contains only encoded slices

ais object show is an for ais object show - both can be used interchangeably.

Show default object properties

Display default properties of object list.txt from bucket texts:

1 $ ais object show ais://texts/list.txt
2 PROPERTY    VALUE
3 checksum    2d61e9b8b299c41f
4 size        7.63MiB
5 atime       06 Jan 20 14:55 PST
6 version     1

Show all object properties

Display all properties of object list.txt from bucket texts:

1 $ ais object show ais://texts/list.txt --props=all
2 PROPERTY    VALUE
3 atime       06 Jan 20 14:55 PST
4 checksum    2d61e9b8b299c41f
5 copies      1 [/data/mp1]
6 custom      -
7 ec          1:1[replicated]
8 name        provider://texts/list.txt
9 node        t[neft8086]
10 size        7.63MiB
11 version     2

Show selected object properties

Show only selected (size,version,ec) properties:

1 $ ais object show --props size,version,ec ais://texts/listx.txt
2 PROPERTY    VALUE
3 size        7.63MiB
4 version     1
5 ec          2:2[replicated]

PUT object

Briefly:

ais put [-|FILE|DIRECTORY[/PATTERN]] BUCKET[/OBJECT_NAME_or_PREFIX]¹ [command options]

writes a single file, an entire directory (of files), or a typed content directly from STDIN (-) - into the specified (destination) bucket.

Notice the optional [/PATTERN] - a regular shell filename-matching primitive - to select files from the source directory.

If an object of the same name exists, the object will be overwritten without confirmation

but only if is different, content-wise - writing identical bits is optimized-out

If CLI detects that a user is going to put more than one file, it calculates the total number of files, total data size, and checks if the bucket is empty.

Then it shows all gathered info to the user and asks for confirmation to continue.

Confirmation request can be disabled with the option --yes for use in scripts.

When writing from STDIN, type Ctrl-D to terminate the input.

Inline help

1 $ ais put --help
2 
3 NAME:
4    ais put - (alias for "object put") PUT or append one file, one directory, or multiple files and/or directories.
5    Use optional shell filename PATTERN (wildcard) to match/select multiple sources.
6    Destination naming is consistent with 'ais object promote' command, whereby the optional OBJECT_NAME_or_PREFIX
7    becomes either a name, a prefix, or a virtual destination directory (if it ends with a forward '/').
8    Assorted examples and usage options follow (and see docs/cli/object.md for more):
9      - upload matching files: 'ais put "docs/*.md" ais://abc/markdown/'
10      - (notice quotation marks and a forward slash after 'markdown/' destination);
11      - '--compute-checksum': use '--compute-checksum' to facilitate end-to-end protection;
12      - '--progress': progress bar, to show running counts and sizes of uploaded files;
13      - Ctrl-D: when writing directly from standard input use Ctrl-D to terminate;
14      - '--append' to append (concatenate) files, e.g.: 'ais put docs ais://nnn/all-docs --append';
15      - '--dry-run': see the results without making any changes.
16      Notes:
17      - to write or add files to (.tar, .tgz or .tar.gz, .zip, .tar.lz4)-formatted objects ("shards"), use 'ais archive'
18 
19 USAGE:
20    ais put [-|FILE|DIRECTORY[/PATTERN]] BUCKET[/OBJECT_NAME_or_PREFIX] [command options]
21 
22 OPTIONS:
23    --append             Concatenate files: append a file or multiple files as a new _or_ to an existing object
24    --chunk-size value   Chunk size in IEC or SI units, or "raw" bytes (e.g.: 4mb, 1MiB, 1048576, 128k; see '--units')
25    --compute-checksum   Compute client-side checksum - one of the supported checksum types that is currently configured for the destination bucket -
26                         and provide it as part of the PUT request for subsequent validation on the server side
27                         (see also: "end-to-end protection")
28    --cont-on-err        Keep running archiving xaction (job) in presence of errors in a any given multi-object transaction
29    --crc32c value       compute client-side crc32c checksum
30                         and provide it as part of the PUT request for subsequent validation on the server side
31    --dry-run            Preview the results without really running the action
32    --include-src-dir    Prefix destination object names with the source directory
33    --list value         Comma-separated list of object or file names, e.g.:
34                         --list 'o1,o2,o3'
35                         --list "abc/1.tar, abc/1.cls, abc/1.jpeg"
36                         or, when listing files and/or directories:
37                         --list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
38    --md5 value          compute client-side md5 checksum
39                         and provide it as part of the PUT request for subsequent validation on the server side
40    --num-workers value  Number of concurrent client-side workers (to execute PUT or append requests);
41                         use (-1) to indicate single-threaded serial execution (ie., no workers);
42                         any positive value will be adjusted _not_ to exceed twice the number of client CPUs (default: 10)
43    --progress           Show progress bar(s) and progress of execution in real time
44    --recursive, -r      Recursive operation
45    --refresh value      Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
46                         valid time units: ns, us (or µs), ms, s (default), m, h
47    --retries value      When failing to PUT retry the operation up to so many times (with increasing timeout if timed out) (default: 1)
48    --sha256 value       compute client-side sha256 checksum
49                         and provide it as part of the PUT request for subsequent validation on the server side
50    --sha512 value       compute client-side sha512 checksum
51                         and provide it as part of the PUT request for subsequent validation on the server side
52    --skip-lookup        Do not execute HEAD(bucket) request to lookup remote bucket and its properties; possible usage scenarios include:
53                          1) adding remote bucket to aistore without first checking the bucket's accessibility
54                             (e.g., to configure the bucket's aistore properties with alternative security profile and/or endpoint)
55                          2) listing public-access Cloud buckets where certain operations (e.g., 'HEAD(bucket)') may be disallowed
56    --skip-vc            Skip loading object metadata (and the associated checksum & version related processing)
57    --template value     Template to match object or file names; may contain prefix (that could be empty) with zero or more ranges
58                         (with optional steps and gaps), e.g.:
59                         --template "" # (an empty or '*' template matches everything)
60                         --template 'dir/subdir/'
61                         --template 'shard-{1000..9999}.tar'
62                         --template "prefix-{0010..0013..2}-gap-{1..2}-suffix"
63                         and similarly, when specifying files and directories:
64                         --template '/home/dir/subdir/'
65                         --template "/abc/prefix-{0010..9999..2}-suffix"
66    --timeout value      Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
67                         valid time units: ns, us (or µs), ms, s (default), m, h
68    --units value        Show statistics and/or parse command-line specified sizes using one of the following units of measurement:
69                         iec - IEC format, e.g.: KiB, MiB, GiB (default)
70                         si  - SI (metric) format, e.g.: KB, MB, GB
71    --verbose, -v        Verbose output
72    --wait               Wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
73    --xxhash value       compute client-side xxhash checksum
74                         and provide it as part of the PUT request for subsequent validation on the server side
75    --yes, -y            Assume 'yes' to all questions
76    --help, -h           Show help

1 FILE|DIRECTORY should point to a file or a directory. Wildcards are supported, but they work a bit differently from shell wildcards. Symbols * and ? can be used only in a file name pattern. Directory names cannot include wildcards. Only a file name is matched, not full file path, so /home/user/*.tar --recursive matches not only .tar files inside /home/user but any .tar file in any /home/user/ subdirectory. This makes shell wildcards like ** redundant, and the following patterns won’t work in ais: /home/user/img-set-*/*.tar or /home/user/bck/**/*.tar.gz

FILE must point to an existing file. File masks and directory uploading are NOT supported in single-file upload mode.

Object names

PUT command handles two possible ways to specify resulting object name if source references single file:

Object name is not provided: ais put path/to/(..)/file.go bucket/ creates object file.go in bucket
Explicit object name is provided: ais put path/to/(..)/file.go bucket/path/to/object.go creates object path/to/object.go in bucket

PUT command handles object naming with range syntax as follows:

Object names are file paths without longest common prefix of all files from source. This means that the leading part of file path until the last / before first { is excluded from object name.
OBJECT_NAME is prepended to each object name.
Abbreviations in source like ../ are not supported at the moment.

PUT command handles object naming if its source references directories:

For path p of source directory, resulting objects names are path to files with trimmed p prefix
OBJECT_NAME is prepended to each object name.
Abbreviations in source like ../ are not supported at the moment.

All examples below put into an empty bucket and the source directory structure is:

/home/user/bck/img1.tar
/home/user/bck/img2.zip
/home/user/bck/extra/img1.tar
/home/user/bck/extra/img3.zip

The current user HOME directory is /home/user.

Put with client-side checksumming

Motivation: There’s always a motivation to perform faster. One way to achieve this is by avoiding redundant writes of user data. A write operation can effectively become a no-op if the identical data already exists in the cluster. The conventional method to establish such identity is through content checksumming.

In short, here’s a CLI write-optimizing trick that utilizes client-side checksumming.

First PUT:

1 $ time ais put /tmp/www s3://ais-aa --yes --compute-checksum --recursive
2 
3 Files to upload:
4 EXTENSION        COUNT   SIZE
5                  27      562.90MiB
6 .go              1       123B
7 .prev            1       7.62MiB
8 .txt             2       10.87KiB
9 TOTAL            31      570.53MiB
10 Uploaded 4(12%) objects, 9.6MiB (1%).
11 Uploaded 8(25%) objects, 55.3MiB (9%).
12 Uploaded 13(41%) objects, 105.8MiB (18%).
13 Uploaded 23(74%) objects, 315.7MiB (55%).
14 Uploaded 29(93%) objects, 449.3MiB (78%).
15 Uploaded 31(100%) objects, 570.5MiB (100%).
16 
17 PUT 31 files (one directory, recursively) => s3://ais-aa
18 
19 real    0m44.895s  <<<<<<<<<<<<<<<<<<<<<< 45s
20 user    0m0.097s
21 sys     0m0.355s

Second PUT with no changes at the source:

1 $ time ais put /tmp/www s3://ais-aa --yes --compute-checksum --recursive
2 
3 Files to upload:
4 EXTENSION        COUNT   SIZE
5                  27      562.90MiB
6 .go              1       123B
7 .prev            1       7.62MiB
8 .txt             2       10.87KiB
9 TOTAL            31      570.53MiB
10 
11 PUT 31 files (one directory, recursively) => s3://ais-aa
12 
13 real    0m0.136s   <<<<<<<<<<<<<<<<<<<<<<<<<<<< (PUT took no time)
14 user    0m0.107s
15 sys     0m0.509s

Adding one file to the source:

1 $ time ais put /tmp/www s3://ais-aa --yes --compute-checksum --recursive
2 
3 Files to upload:
4 EXTENSION        COUNT   SIZE
5                  28      563.17MiB
6 .go              1       123B
7 .prev            1       7.62MiB
8 .txt             2       10.87KiB
9 TOTAL            32      570.80MiB
10 
11 PUT 32 files (one directory, recursively) => s3://ais-aa
12 
13 real    0m1.029s   <<<<<<<<<<<<<<<<<< 1s
14 user    0m0.121s
15 sys     0m0.588s

Note: Ideally, the checksum is provided with PUT API calls. The CLI takes it one step further: if client-side checksumming is requested but the checksum is empty, the CLI computes it automatically. The corresponding overhead must be taken into account when analyzing resulting performance.

Put single file

First, compare two simple examples:

1 $ ais put README.md ais://nnn/ccc/
2 PUT "README.md" => ais://nnn/ccc/README.md
3 
4 $ ais put README.md ais://nnn/ccc
5 PUT "README.md" => ais://nnn/ccc

In other words, a trailing forward slash in the destination name is interpreted as a destination directory

which is what one would expect from something like Bash: cp README.md /nnn/ccc/

One other example: put a single file img1.tar into local bucket mybucket, name it img-set-1.tar.

1 $ ais put "/home/user/bck/img1.tar" ais://mybucket/img-set-1.tar
2 # PUT /home/user/bck/img1.tar => ais://mybucket/img-set-1.tar

Put single file with checksum

Put a single file img1.tar into local bucket mybucket, with a content checksum flag to override the default bucket checksum performed at the server side.

1 $ ais put "/home/user/bck/img1.tar" ais://mybucket/img-set-1.tar --crc32c 0767345f
2 # PUT /home/user/bck/img1.tar => ais://mybucket/img-set-1.tar
3 
4 $ ais put "/home/user/bck/img1.tar" ais://mybucket/img-set-1.tar --md5 e91753513c7fc873467c1f3ca061fa70
5 # PUT /home/user/bck/img1.tar => ais://mybucket/img-set-1.tar
6 
7 $ ais put "/home/user/bck/img1.tar" ais://mybucket/img-set-1.tar --sha256 dc2bac3ba773b7bc52c20aa85e6ce3ae097dec870e7b9bda03671a1c434b7a5d
8 # PUT /home/user/bck/img1.tar => ais://mybucket/img-set-1.tar
9 
10 $ ais put "/home/user/bck/img1.tar" ais://mybucket/img-set-1.tar --sha512 e7da5269d4cd882deb8d7b7ca5cbf424047f56815fd7723123482e2931823a68d866627a449a55ca3a18f9c9ba7c8bb6219a028ba3ff5a5e905240907d087e40
11 # PUT /home/user/bck/img1.tar => ais://mybucket/img-set-1.tar
12 
13 $ ais put "/home/user/bck/img1.tar" ais://mybucket/img-set-1.tar --xxhash 05967d5390ac53b0
14 # PUT /home/user/bck/img1.tar => ais://mybucket/img-set-1.tar

Optionally, the user can choose to provide a --compute-cksum flag for the checksum flag and let the API take care of the computation.

1 $ ais put "/home/user/bck/img1.tar" ais://mybucket/img-set-1.tar --compute-cksum
2 # PUT /home/user/bck/img1.tar => ais://mybucket/img-set-1.tar

Put single file with implicitly defined name

Put a single file ~/bck/img1.tar into bucket mybucket, without explicit name.

1 $ ais put "~/bck/img1.tar" ais://mybucket/
2 
3 # PUT /home/user/bck/img1.tar => mybucket/img-set-1.tar

Put content from STDIN

Read unpacked content from STDIN and put it into bucket mybucket with name img-unpacked.

Note that content is put in chunks that can have a slight overhead. --chunk-size allows for controlling the chunk size - the bigger the chunk size the better performance (but also higher memory usage).

$ $ tar -xOzf ~/bck/img1.tar | ais put - ais://mybucket/img1-unpacked
$ # PUT /home/user/bck/img1.tar (as stdin) => ais://mybucket/img-unpacked

Put directory

Put two objects, /home/user/bck/img1.tar and /home/user/bck/img2.zip, into the root of bucket mybucket. Note that the path /home/user/bck is a shortcut for /home/user/bck/* and that recursion is disabled by default.

1 $ ais put "/home/user/bck" ais://mybucket
2 
3 # PUT /home/user/bck/img1.tar => img1.tar
4 # PUT /home/user/bck/img2.tar => img2.zip

Alternatively, to reference source directory we can use relative (../..) naming.

Also notice progress bar (the --progress flag) and g* wildcard that allows to select only the filenames that start with ‘g’

1 $ ais put "../../../../bin/g*" ais://vvv --progress
2 Files to upload:
3 EXTENSION        COUNT   SIZE
4                  8       99.82MiB
5 .0               2       46.28MiB
6 TOTAL           10      146.10MiB
7 Proceed putting to ais://vvv? [Y/N]: y
8 Uploaded files progress                   10/10 [==============================================================] 100 %
9 Uploaded sizes progress 146.10 MiB / 146.10 MiB [==============================================================] 100 %
10 PUT 10 objects to "ais://vvv"

NOTE double quotes to denote the "../../../../bin/g*" source above. With pattern matching, using quotation marks is a MUST. Single quotes can be used as well.

Put multiple files with prefix added to destination object names

The multi-file source can be: a directory, a comma-separated list, a template-defined range - all of the above.

Examples follow below, but also notice:

the flexibility in terms specifying source-matching templates, and
destination prefix - with or without trailing forward slash

Example 1.

1 $ ais put ais://nnn/fff --template "/tmp/www/shard-{001..002}.tar"
2 Warning: 'fff' will be used as the destination name prefix for all files matching '/tmp/www/shard-{001..002}.tar'
3 Proceed anyway? [Y/N]: y
4 Files to upload:
5 EXTENSION        COUNT   SIZE
6 .tar             2       17.00KiB
7 TOTAL            2       17.00KiB
8 PUT 2 files => ais://nnn/fff? [Y/N]: y
9 Done
10 $ ais ls ais://nnn
11 NAME                     SIZE
12 fffshard-001.tar         8.50KiB
13 fffshard-002.tar         8.50KiB

Example 2.

Same as above, except now we make sure that destination is a virtual directory (notice trailing forward ’/’):

1 $ ais put ais://nnn/ggg/ --template "/tmp/www/shard-{003..004}.tar"
2 Files to upload:
3 EXTENSION        COUNT   SIZE
4 .tar             2       17.00KiB
5 TOTAL            2       17.00KiB
6 PUT 2 files => ais://nnn/ggg/? [Y/N]: y
7 Done
8 $ ais ls ais://nnn
9 NAME                     SIZE
10 fffshard-001.tar         8.50KiB
11 fffshard-002.tar         8.50KiB
12 ggg/shard-003.tar        8.50KiB
13 ggg/shard-004.tar        8.50KiB

Example 3.

Same as above, with --template embedded into the source argument:

1 $ ais put "/tmp/www/shard-{005..006}.tar"  ais://nnn/hhh/
2 Files to upload:
3 EXTENSION        COUNT   SIZE
4 .tar             2       17.00KiB
5 TOTAL            2       17.00KiB
6 PUT 2 files => ais://nnn/hhh/? [Y/N]: y
7 Done
8 $ ais ls ais://nnn
9 NAME                     SIZE
10 fffshard-001.tar         8.50KiB
11 fffshard-002.tar         8.50KiB
12 ggg/shard-003.tar        8.50KiB
13 ggg/shard-004.tar        8.50KiB
14 hhh/shard-005.tar        8.50KiB
15 hhh/shard-006.tar        8.50KiB

And finally, we can certainly PUT source directory:

Example 4.

1 $ ais put /home/user/bck ais://mybucket/subdir/
2 
3 # PUT /home/user/bck/img1.tar => ais://mybucket/subdir/img1.tar
4 # PUT /home/user/bck/img2.tar => ais://mybucket/subdir/img2.zip
5 # PUT /home/user/bck/extra/img1.tar => ais://mybucket/subdir/extra/img1.tar
6 # PUT /home/user/bck/extra/img3.zip => ais://mybucket/subdir/extra/img3.zip

The same as above, but without trailing /.

1 $ ais put "/home/user/bck" ais://mybucket/subdir
2 
3 # PUT /home/user/bck/img1.tar => ais://mybucket/subdirimg1.tar
4 # PUT /home/user/bck/img2.tar => ais://mybucket/subdirimg2.zip
5 # PUT /home/user/bck/extra/img1.tar => ais://mybucket/subdirextra/img1.tar
6 # PUT /home/user/bck/extra/img3.zip => ais://mybucket/subdirextra/img3.zip

Put multiple files into virtual directory, track progress

Same as above with source files in double quotes below, and with progress bar:

List of sources that you want to upload can (a) comprize any number (and any mix) of comma-separated files and/or directories, and (b) must be embedded in double or single quotes.

1 $ ais put "README.md, LICENSE" ais://aaa/my-virt-dir/ --progress -y
2 Files to upload:
3 EXTENSION        COUNT   SIZE
4                  1       1.05KiB
5 .md              1       11.24KiB
6 TOTAL            2       12.29KiB
7 Uploaded files:                   2/2 [==============================================================] 100 %
8 Total size:     12.29 KiB / 12.29 KiB [==============================================================] 100 %
9 PUT 2 objects (non-recursive) to "ais://aaa"

Note ’/’ suffix in my-virt-dir/ above - without trailing filepath separator we would simply get a longer filename (filenames) at the root of the destination bucket.

We can now list them in the bucket ais://aaa the way we would list a directory:

1 $ ais ls ais://aaa --prefix my-virt-dir
2 NAME                     SIZE
3 my-virt-dir/LICENSE      1.05KiB
4 my-virt-dir/README.md    11.24KiB

Put pattern-matching files from directory

Same as above, except that only files matching pattern *.tar are PUT, so the final bucket content is tars/img1.tar and tars/extra/img1.tar.

NOTE double quotes to denote the source. With pattern matching, using quotation marks is a MUST. Single quotes can be used as well.

1 $ ais put "~/bck/*.tar" ais://mybucket/tars/
2 # PUT /home/user/bck/img1.tar => ais://mybucket/tars/img1.tar
3 # PUT /home/user/bck/extra/img1.tar => ais://mybucket/tars/extra/img1.tar

Same as above with progress bar, recursion into nested directories, and matching characters anywhere in the filename:

1 $ ais put "ais/*_t*" ais://vvv --progress --recursive
2 Files to upload:
3 EXTENSION        COUNT   SIZE
4 .go              43      704.40KiB
5 TOTAL            43      704.40KiB
6 PUT 43 files => ais://vvv? [Y/N]: y
7 
8 Uploaded files progress                   43/43 [==============================================================] 100 %
9 Uploaded sizes progress 704.40 KiB / 704.40 KiB [==============================================================] 100 %
10 PUT 43 objects to "ais://vvv"

The result will look as follows:

1 ...
2 test/target_test.go              1.55KiB
3 test/various_test.go             510B
4 test/xaction_test.go             2.61KiB
5 tgtobj_test.go                   5.57KiB
6 utils_test.go                    1.38KiB

Put a range of files

There are several equivalent ways to PUT a templated range of files:

Example 1.

Put 9 files to mybucket using a range request. Note the formatting of object names. They exclude the longest parent directory of path which doesn’t contain a template ({a..b}).

$ $ for d1 in {0..2}; do for d2 in {0..2}; do echo "0" > ~/dir/test${d1}${d2}.txt; done; done
$ 
$ # NOTE: make sure to use double or sinle quotes around the range
$ 
$ $ ais put "~/dir/test{0..2}{0..2}.txt" ais://mybucket -y
$ 9 objects put into "ais://mybucket" bucket

Example 2. PUT a range of files into a virtual directory

Same as above but in addition destination object names will have additional prefix subdir/ (notice the trailing /)

In other words, this PUT in affect creates a virtual directory inside destination ais://mybucket

$ # first, prepare test files
$ $ for d1 in {0..2}; do for d2 in {0..2}; do echo "0" > ~/dir/test${d1}${d2}.txt; done; done

Next, PUT:

1 $ ais put "~/dir/test{0..2}{0..2}.txt" ais://mybucket/subdir/ -y

Example 3.

Finally, the same exact operation can be accomplished using --template option

--template is universally supported to specify a range of files or objects

1 $ ais put ais://mybucket/dir/ -y --template "~/dir/test{0..2}{0..2}.txt"

Put a list of files

There are several equivalent ways to PUT a list of files:

Example 1. Notice the double quotes (single quotes can be used as well)

1 $ ais put "README.md,LICENSE" s3://abc
2 Files to upload:
3 EXTENSION        COUNT   SIZE
4                  1       1.05KiB
5 .md              1       11.24KiB
6 TOTAL            2       12.29KiB
7 PUT 2 files => s3://abc? [Y/N]: y

Example 2.

Alternatively, the same can be done using the --list flag:

--list is universally supported to specify a list of files or objects

1 $ ais put s3://abc --list "README.md,LICENSE"
2 
3 Files to upload:
4 EXTENSION        COUNT   SIZE
5                  1       1.05KiB
6 .md              1       11.24KiB
7 TOTAL            2       12.29KiB
8 PUT 2 files => s3://abc? [Y/N]: y

Example 3. PUT a list into virtual directory

The only difference from the two examples above is: trailing / in the destination name.

1 $ ais put ais://abc/subdir/ --list 'LICENSE,README.md' -y
2 
3 $ ais ls ais://abc
4 NAME                     SIZE
5 subdir/LICENSE           1.05KiB
6 subdir/README.md         11.24KiB

Dry-Run option

Preview the files that would be sent to the cluster, without actually putting them.

Example 1

$ $ for d1 in {0..2}; do for d2 in {0..2}; mkdir -p ~/dir/test${d1}/dir && do echo "0" > ~/dir/test${d1}/dir/test${d2}.txt; done; done
$ $ ais put "~/dir/test{0..2}/dir/test{0..2}.txt" ais://mybucket --dry-run
$ 
$ [DRY RUN] No modifications on the cluster
$ /home/user/dir/test0/dir/test0.txt => ais://mybucket/test0/dir/test0.txt
$ (...)

Example 2

Generally, the --template option combines (an optional) prefix and/or one or more ranges (e.g., bash brace expansions).

In this example, we only use the “prefix” part of the --template to specify source directory.

1 $ ls -l /tmp/w
2 total 32
3 -rw-r--r-- 1 root root 14180 Dec 11 18:18 111
4 -rw-r--r-- 1 root root 14180 Dec 11 18:18 222
5 
6 $ ais put ais://nnn/fff --template /tmp/w --dry-run
7 [DRY RUN] with no modifications to the cluster
8 Warning: 'fff' will be used as the destination name prefix for all files from '/tmp/w' directory
9 Proceed anyway? [Y/N]: y
10 Files to upload:
11 EXTENSION        COUNT   SIZE
12                  2       27.70KiB
13 TOTAL            2       27.70KiB
14 [DRY RUN] PUT 2 files (one directory, non-recursive) => ais://nnn/fff
15 PUT /tmp/w/222 -> ais://nnn/fff222
16 PUT /tmp/w/111 -> ais://nnn/fff111

Note: to PUT files into a virtual destination directory, use trailing ’/’, e.g.: ais put ais://nnn/fff/ ...

Put multiple directories using Bash range notation

First, let’s generate some files and directories (strictly for illustration purposes):

$ $ for d1 in {0..10}; do mkdir /tmp/testdir_$d1 && for d2 in {0..2}; do echo "0" > /tmp/testdir_$d1/test${d2}.txt; done; done

Next, PUT them all in one shot (notice quotation marks!):

$ $ ais put "/tmp/testdir_{0..10}" ais://nnn
$ Files to upload:
$ EXTENSION        COUNT   SIZE
$ .txt             33      66B
$ TOTAL            33      66B
$ 
$ PUT 33 files (11 directories, non-recursive) => ais://nnn? [Y/N]:

Let’s now take a look at the result - and observe a PROBLEM:

1 $ ais ls ais://nnn --summary
2 NAME             PRESENT         OBJECTS         SIZE (apparent, objects, remote)        USAGE(%)
3 ais://nnn        yes             3 0             112.01KiB 6B 0B                         0%

So Yes, the problem is that by default destination object names are sourced from the source file basenames.

In this examples, we happen to have only 3 basenames: test0.txt, test1.txt, and test2.txt.

The workaround is to include respective parent directories in the destination naming:

As always, see ais put --help for usage examples and more options.

1 $ ais put "/tmp/testdir_{0..10}" ais://nnn --include-src-dir
2 Files to upload:
3 EXTENSION        COUNT   SIZE
4 .txt             33      66B
5 TOTAL            33      66B
6 
7 PUT 33 files (11 directories, non-recursive) => ais://nnn? [Y/N]: y
8 Done
9 
10 $ ais ls ais://nnn --summary
11 NAME             PRESENT         OBJECTS         SIZE (apparent, objects, remote)        USAGE(%)
12 ais://nnn        yes             33 0            320.06KiB 66B 0B                        0%

Put multiple directories using filename-matching pattern (wildcard)

Same as above, but note: alternative syntax, which is maybe more conventional:

$ $ ais put "/tmp/testdir_*" ais://nnn --include-src-dir
$ Files to upload:
$ EXTENSION        COUNT   SIZE
$ .txt             33      66B
$ TOTAL            33      66B
$ 
$ PUT 33 files (11 directories, non-recursive) => ais://nnn? [Y/N]:

Put multiple directories with the `--skip-vc` option

The --skip-vc option allows AIS to skip loading existing object’s metadata to perform metadata-associated processing (such as comparing source and destination checksums, for instance). In certain scenarios (e.g., massive uploading of new files that cannot be present in the bucket) this can help reduce PUT latency.

$ ## prepare testing content
$ $ for d1 in {0..10}; do mkdir /tmp/testdir_$d1 && for d2 in {0..2}; do echo "0" > /tmp/testdir_$d1/test${d2}.txt; done; done
$ 
$ ## PUT
$ $ ais put ""/tmp/testdir_{0..10}"" ais://mybucket -y --skip-vc
$ 
$ Files to upload:
$ EXTENSION        COUNT   SIZE
$ .txt             33      66B
$ TOTAL            33      66B

Tips for Copying Files from Lustre (NFS)

Yes, ais put can be used to copy remote files - usage tips follow below. Buf first, disclaimer.

Disclaimer

Copying large amounts of data from remote (NFS, SMB) locations is not exactly an exercise for a single client machine. There are alternative designed-in ways, whereby all AIStore nodes partition remote source between themselves and do the copying - in parallel.

Performance-wise, the difference from copying via client (or by client) - is two-fold:

many orders of magnitude greater horsepower that AIStore can contribute to the effort, and
avoidance of the (client <= NFS) and (client => AIStore) roundtrips.

Needless to say, promoting files to objects, as it were, requires that all AIS nodes have connectivity and permissions to access the remote source.

Further references:

Tips

Use --retries option

Including --retries in your command will help resolve an occasional timeout and other intermittent failures. For example, --retries 5 will retry a failed requests up to 5 (five) times.

1 $ ais put --help
2 ...
3 
4    --retries value      when failing to PUT retry the operation up to so many times (with increasing timeout if timed out) (default: 1)

Use --num-workers option

In other words, take advantage of the client side multi-threading. If you have sufficient resources, increase this number to allow more workers to transfer data in parallel.

1 $ ais put --help
2 ...
3 
4    --num-workers value  number of concurrent client-side workers (to execute PUT or append requests);
5                         use (-1) to indicate single-threaded serial execution (ie., no workers);
6                         any positive value will be adjusted _not_ to exceed twice the number of client CPUs (default: 10)

Example 1

Recursively copy the contents of (NFS-mounted) target_dir/ to the ais://nnn/target_dir/ bucket, using 64 client workers (OS threads) and retrying failed requests up to 3 times.

1 $ ais object put -r -y --num-workers 64 --retries 3 target_dir/ ais://nnn/target_dir/

Example 2

Same as above (and notice ais put shortcut and --include-src-dir option):

1 $ ais put target_dir ais://nnn -r -y --num-workers 64 --retries 3 --include-src-dir

Example 3

Same as above, but with additional capability to “continue on error” - skip errors that may arise when traversing the source tree:

1 $ ais put target_dir ais://nnn --recursive --yes --num-workers 64 --retries 3 --include-src-dir --cont-on-err

Example 4

Same as above, but in addition ask CLI to report all errors that may be skipped or ignored due to the --cont-on-err flag:

1 $ ais config cli verbose
2 PROPERTY         VALUE
3 verbose          false
4 
5 $ ais config cli set verbose true
6 "verbose" set to: "true" (was: "false")
7 
8 $ ais put target_dir ais://nnn --recursive --yes --num-workers 64 --retries 3 --include-src-dir --cont-on-err

Patience

Be patient: copying from remote locations is subject to network and remote servers’ delays, both.

Also and separately, note that at the time of this writing AIS CLI does not support pagination of the remote directories that may contain millions of entries. Listing of the entire remote source is (currently) done in one shot, and prior to copying.

If ais put process seems to have paused, there’s a good chance it is still listing remote files or copying in the background.

Refrain from pressing Ctrl-C to interrupt it.

When your destination bucket is S3 or similar

Waiting time may be even greater if you are copying data to an AIStore s3://, gs://, or az:// bucket. AIS uses write-through, so the same data is written to the remote backend and locally as one atomic transaction.

Finally, try to transition to WebDataset formatting

Copying, or generally, working in any shape and form with many (millions of) small files comes with significant and unavoidable overhead, both networking and storage-wise.

Use our ishard tool to convert and serialize your data using the preferred formatting (a.k.a. WebDataset convention):

Promote files and directories

Inline help follows below:

1 $ ais object promote --help
2 NAME:
3    ais object promote - PROMOTE target-accessible files and directories.
4    The operation is intended for copying NFS and SMB shares mounted on any/all targets
5    but can be also used to copy local files (again, on any/all targets in the cluster).
6    Copied files and directories become regular stored objects that can be further listed and operated upon.
7    Destination naming is consistent with 'ais put' command, e.g.:
8      - 'promote /tmp/subdir/f1 ais://nnn'        - ais://nnn/f1
9      - 'promote /tmp/subdir/f2 ais://nnn/aaa'    - ais://nnn/aaa
10      - 'promote /tmp/subdir/f3 ais://nnn/aaa/'   - ais://nnn/aaa/f3
11      - 'promote /tmp/subdir ais://nnn'           - ais://nnn/f1, ais://nnn/f2, ais://nnn/f3
12      - 'promote /tmp/subdir ais://nnn/aaa/'      - ais://nnn/aaa/f1, ais://nnn/aaa/f2, ais://nnn/aaa/f3
13    Other supported options follow below.
14 
15 USAGE:
16    ais object promote FILE|DIRECTORY[/PATTERN] BUCKET[/OBJECT_NAME_or_PREFIX] [command options]
17 
18 OPTIONS:
19    --recursive, -r      recursive operation
20    --overwrite-dst, -o  overwrite destination, if exists
21    --not-file-share     each target must act autonomously skipping file-share auto-detection and promoting the entire source (as seen from the target)
22    --delete-src         delete successfully promoted source
23    --target-id value    ais target designated to carry out the entire operation
24    --verbose, -v        verbose output
25    --help, -h           show help

Options

1 $ ais object promote --help
2 
3 NAME:
4    ais object promote - PROMOTE target-accessible files and directories.
5    The operation is intended for copying NFS and SMB shares mounted on any/all targets
6    but can be also used to copy local files (again, on any/all targets in the cluster).
7    Copied files and directories become regular stored objects that can be further listed and operated upon.
8    Destination naming is consistent with 'ais put' command, e.g.:
9      - 'promote /tmp/subdir/f1 ais://nnn'        - ais://nnn/f1
10      - 'promote /tmp/subdir/f2 ais://nnn/aaa'    - ais://nnn/aaa
11      - 'promote /tmp/subdir/f3 ais://nnn/aaa/'   - ais://nnn/aaa/f3
12      - 'promote /tmp/subdir ais://nnn'           - ais://nnn/f1, ais://nnn/f2, ais://nnn/f3
13      - 'promote /tmp/subdir ais://nnn/aaa/'      - ais://nnn/aaa/f1, ais://nnn/aaa/f2, ais://nnn/aaa/f3
14    Other supported options follow below.
15 
16 USAGE:
17    ais object promote FILE|DIRECTORY[/PATTERN] BUCKET[/OBJECT_NAME_or_PREFIX] [command options]
18 
19 OPTIONS:
20    --delete-src         Delete successfully promoted source
21    --not-file-share     Each target must act autonomously skipping file-share auto-detection and promoting the entire source (as seen from the target)
22    --overwrite-dst, -o  Overwrite destination, if exists
23    --recursive, -r      Recursive operation
24    --skip-lookup        Do not execute HEAD(bucket) request to lookup remote bucket and its properties; possible usage scenarios include:
25                          1) adding remote bucket to aistore without first checking the bucket's accessibility
26                             (e.g., to configure the bucket's aistore properties with alternative security profile and/or endpoint)
27                          2) listing public-access Cloud buckets where certain operations (e.g., 'HEAD(bucket)') may be disallowed
28    --target-id value    AIS target designated to carry out the entire operation
29    --verbose, -v        Verbose output
30    --help, -h           Show help

Destination naming

See above.

Promote a single file

Promote /tmp/examples/example1.txt without specified object name.

1 $ ais object promote /tmp/examples/example1.txt ais://mybucket --keep=true
2 # PROMOTE /tmp/examples/example1.txt => ais://mybucket/example1.txt

Promote file while specifying custom (resulting) name

Promote /tmp/examples/example1.txt as object with name example1.txt.

1 $ ais object promote /tmp/examples/example1.txt ais://mybucket/example1.txt --keep=true
2 # PROMOTE /tmp/examples/example1.txt => ais://mybucket/example1.txt

Promote a directory

Make AIS objects out of /tmp/examples files (one file = one object). /tmp/examples is a directory present on some (or all) of the deployed storage nodes.

1 $ ais object promote /tmp/examples ais://mybucket/ -r --keep=true

Promote directory with custom prefix

Promote /tmp/examples files to AIS objects. Object names will have examples/ prefix.

1 $ ais object promote /tmp/examples ais://mybucket/examples/ -r --keep=false

Promote invalid path

Try to promote a file that does not exist.

1 $ ais create ais://testbucket
2 "ais://testbucket" bucket created
3 $ ais show cluster
4 TARGET          MEM USED %  MEM AVAIL   CAP USED %  CAP AVAIL   CPU USED %  REBALANCE
5 1014646t8081    0.00%	    4.00GiB	59%         375.026GiB  0.00%	    finished
6 ...
7 $ ais object promote /target/1014646t8081/nonexistent/dir/ ais://testbucket --target 1014646t8081 --keep=false
8 (...) Bad Request: stat /target/1014646t8081/nonexistent/dir: no such file or directory

Multipart upload

ais object multipart-upload or, same, ais object mpu - Upload large objects in multiple parts for improved performance and reliability.

Multipart upload allows you to upload large objects by breaking them into smaller, manageable parts. This provides several benefits:

Improved performance: Parts can be uploaded in parallel
Reliability: Failed part uploads can be retried without affecting other parts
Flexibility: Parts can be uploaded in any order
Resume capability: Ability to abort and restart uploads

The multipart upload process consists of three main steps:

Create a multipart upload session to get an upload ID
Upload parts using the upload ID (parts can be uploaded in parallel)
Complete the upload by assembling all parts into the final object

Options

1 $ ais mpu --help
2 
3 NAME:
4    ais mpu - (alias for "object multipart-upload") Multipart upload operations: create, put-part, complete, and abort
5 
6 USAGE:
7    ais mpu command [arguments...]  [command options]
8 
9 COMMANDS:
10    create  Create a multipart upload session for large objects.
11       Returns an UPLOAD_ID that must be used for subsequent part uploads and completion, e.g.:
12              - 'mpu create ais://bck/large.dat'            - create MPU session for 'large.dat';
13              - 'mpu create ais://bck/video.mp4 --verbose'  - create multipart upload with verbose output.
14    put-part  Upload individual parts for a multipart upload session with a given UPLOAD_ID (returned by 'mpu create').
15       Parts can be uploaded in parallel and in any order, e.g.:
16         - 'mpu put-part ais://bck/large UPLOAD_ID 2 /path/part2.dat --verbose'                  - upload part 2 with progress;
17         - 'mpu put-part ais://bck/large UPLOAD_ID 1 /path/part1.dat'                            - upload part 1;
18         - 'mpu put-part ais://bck/large --upload-id UPLOAD_ID --part-number 3 /path/part3.dat'  - using flags
19       (for UPLOAD_ID use the value previously returned by 'mpu create' command).
20    complete  Complete a multipart upload by assembling all uploaded parts into the final object.
21       Parts are assembled in the order specified by part numbers, e.g.:
22         - 'mpu complete ais://bck/large UPLOAD_ID 1,2,3,4,5'                         - assemble 5 parts in order;
23         - 'mpu complete ais://bck/large --upload-id UPLOAD_ID --part-numbers 1,2,3'  - using flags;
24         - 'mpu complete ais://bck/large UPLOAD_ID "1,2,3" --verbose'                 - with completion progress
25       (for UPLOAD_ID use the value previously returned by 'mpu create' command).
26    abort  Abort a multipart upload session and clean up any uploaded parts.
27       All uploaded parts are discarded and the object is not created, e.g.:
28         - 'mpu abort ais://bck/large UPLOAD_ID'                        - abort upload session;
29         - 'mpu abort ais://bck/large --upload-id UPLOAD_ID --verbose'  - abort with verbose output
30       (for UPLOAD_ID use the value previously returned by 'mpu create' command).
31 
32 OPTIONS:
33    --help, -h  Show help

Create multipart upload

ais object mpu create BUCKET/OBJECT_NAME

Creates a new multipart upload session and returns an upload ID that must be used for all subsequent operations on this upload.

Create a multipart upload session

1 $ ais object mpu create ais://mybucket/large-video.mp4
2 Upload ID: abc123def456
3 
4 $ ais object mpu create ais://mybucket/large-dataset.tar --verbose
5 Created multipart upload for ais://mybucket/large-dataset.tar
6 Upload ID: xyz789uvw012

Upload parts

ais object mpu put-part BUCKET/OBJECT_NAME UPLOAD_ID PART_NUMBER FILE_PATH

Uploads individual parts for a multipart upload session. Parts can be uploaded in parallel and in any order.

Upload parts sequentially

1 # Upload part 1
2 $ ais object mpu put-part ais://mybucket/large-video.mp4 abc123def456 1 /tmp/video-part1.mp4 --verbose
3 Uploading part 1 from /tmp/video-part1.mp4 (524.29MiB)...
4 Uploaded part 1 for ais://mybucket/large-video.mp4 (upload ID: abc123def456)
5 
6 # Upload part 2
7 $ ais object mpu put-part ais://mybucket/large-video.mp4 abc123def456 2 /tmp/video-part2.mp4 --verbose
8 Uploading part 2 from /tmp/video-part2.mp4 (524.29MiB)...
9 Uploaded part 2 for ais://mybucket/large-video.mp4 (upload ID: abc123def456)
10 
11 # Upload part 3
12 $ ais object mpu put-part ais://mybucket/large-video.mp4 abc123def456 3 /tmp/video-part3.mp4 --verbose
13 Uploading part 3 from /tmp/video-part3.mp4 (451.42MiB)...
14 Uploaded part 3 for ais://mybucket/large-video.mp4 (upload ID: abc123def456)

Upload parts in parallel

Parts can be uploaded simultaneously from different terminals or scripts:

1 # Terminal 1
2 $ ais object mpu put-part ais://mybucket/large-file.dat uploadID 1 /tmp/part1.dat
3 
4 # Terminal 2 (running simultaneously)
5 $ ais object mpu put-part ais://mybucket/large-file.dat uploadID 2 /tmp/part2.dat
6 
7 # Terminal 3 (running simultaneously)
8 $ ais object mpu put-part ais://mybucket/large-file.dat uploadID 3 /tmp/part3.dat

All three commands can be executed at the same time, allowing for faster upload of large files.

Complete multipart upload

ais object mpu complete BUCKET/OBJECT_NAME UPLOAD_ID PART_NUMBERS

Completes a multipart upload by assembling all uploaded parts into the final object. Parts are assembled in the order specified by the part numbers.

Complete upload with all parts

1 $ ais object mpu complete ais://mybucket/large-video.mp4 abc123def456 1,2,3 --verbose
2 Completing multipart upload for ais://mybucket/large-video.mp4 with 3 parts...
3 Successfully completed multipart upload for ais://mybucket/large-video.mp4
4 
5 # Verify the object was created
6 $ ais object ls ais://mybucket --props size
7 NAME                SIZE
8 large-video.mp4     1.50GiB

Complete upload using flags

1 $ ais object mpu complete ais://mybucket/large-dataset.tar --upload-id xyz789uvw012 --part-numbers 1,2,3,4,5
2 Successfully completed multipart upload for ais://mybucket/large-dataset.tar

Complete upload with verbose progress

1 $ ais object mpu complete ais://mybucket/large-file.dat uploadID "1,2,3,4,5,6,7,8" --verbose
2 Completing multipart upload for ais://mybucket/large-file.dat with 8 parts...
3 Successfully completed multipart upload for ais://mybucket/large-file.dat

Abort multipart upload

ais object mpu abort BUCKET/OBJECT_NAME UPLOAD_ID

Aborts a multipart upload session and cleans up any uploaded parts. All uploaded parts are discarded and the object is not created.

Abort an upload session

1 $ ais object mpu abort ais://mybucket/large-video.mp4 abc123def456 --verbose
2 Aborting multipart upload for ais://mybucket/large-video.mp4 (upload ID: abc123def456)...
3 Successfully aborted multipart upload for ais://mybucket/large-video.mp4
4 
5 # Verify the object was not created
6 $ ais object ls ais://mybucket
7 NAME                SIZE
8 # (no large-video.mp4 object)

Example: Complete multipart upload workflow

Here’s a complete example demonstrating the entire multipart upload process:

1 # 1. Create bucket
2 $ ais bucket create ais://mybucket
3 "ais://mybucket" created
4 
5 # 2. Split a large file into parts (example using split command)
6 $ split -b 100M /path/to/large-file.dat /tmp/part-
7 $ ls /tmp/part-*
8 /tmp/part-aa /tmp/part-ab /tmp/part-ac /tmp/part-ad
9 
10 # 3. Create multipart upload session
11 $ ais object mpu create ais://mybucket/large-file.dat
12 Upload ID: mpt123xyz789
13 
14 # 4. Upload all parts
15 $ ais object mpu put-part ais://mybucket/large-file.dat mpt123xyz789 1 /tmp/part-aa --verbose
16 $ ais object mpu put-part ais://mybucket/large-file.dat mpt123xyz789 2 /tmp/part-ab --verbose  
17 $ ais object mpu put-part ais://mybucket/large-file.dat mpt123xyz789 3 /tmp/part-ac --verbose
18 $ ais object mpu put-part ais://mybucket/large-file.dat mpt123xyz789 4 /tmp/part-ad --verbose
19 
20 # 5. Complete the upload
21 $ ais object mpu complete ais://mybucket/large-file.dat mpt123xyz789 1,2,3,4 --verbose
22 Successfully completed multipart upload for ais://mybucket/large-file.dat
23 
24 # 6. Verify the final object
25 $ ais object show ais://mybucket/large-file.dat --props size
26 PROPERTY    VALUE
27 size        400.00MiB

Append object

Append operation (not to confuse with appending or adding to existing archive) can be executed in 3 different ways:

using ais put with --append option;
using ais object concat; and finally
writing from standard input with chunk size (ie., --chunk-size) small enough to require (appending) multiple chunks.

Here’re some examples:

1 ## append all files from a given directory as a single object:
2 
3 $ ais put docs ais://nnn/all-docs --append
4 
5 Created ais://nnn/all-docs (size 571.45KiB)
6 $ ais ls ais://nnn/all-docs -props all
7 PROPERTY         VALUE
8 atime            11 Dec 23 12:18 EST
9 checksum         xxhash[f0eac0698e2489ff]
10 copies           1 [/ais/mp1/7]
11 custom           -
12 ec               -
13 location         t[VQWtTyuI]:mp[/ais/mp1/7, nvme0n1]
14 name             ais://nnn/all-docs
15 size             571.45KiB
16 version          1

1 ## overwrite existing object with 4KiB of random data;
2 ## note that the operation (below) will write about 410 chunks from standard input
3 
4 $ head -c 4096 /dev/urandom | ais object put - ais://nnn/all-docs --chunk-size 10
5 PUT (standard input) => ais://nnn/all-docs
6 
7 $ ais ls ais://nnn/all-docs -props all
8 PROPERTY         VALUE
9 atime            11 Dec 23 12:21 EST
10 checksum         xxhash[b5edf46a1b9459fb]
11 copies           1 [/ais/mp1/7]
12 custom           -
13 ec               -
14 location         t[VQWtTyuI]:mp[/ais/mp1/7, nvme0n1]
15 name             ais://nnn/all-docs
16 size             4.00KiB
17 version          3

Delete object

ais object rm or (same) ais rmo - Delete an object or list/range of objects from a bucket.

1 $ ais rmo --help
2 NAME:
3    ais rmo - (alias for "object rm") Remove object or selected objects from the specified bucket, or buckets - e.g.:
4      - 'rm ais://nnn --all'                                   - remove all objects from the bucket ais://nnn;
5      - 'rm s3://abc' --all                                    - remove all objects including those that are not _present_ in the cluster;
6      - 'rm gs://abc --prefix images/'                         - remove all objects from the virtual subdirectory "images";
7      - 'rm gs://abc/images/'                                  - same as above;
8      - 'rm gs://abc --template images/'                       - same as above;
9      - 'rm gs://abc --template "shard-{0000..9999}.tar.lz4"'  - remove the matching range (prefix + brace expansion);
10      - 'rm "gs://abc/shard-{0000..9999}.tar.lz4"'             - same as above (notice double quotes)
11 
12 USAGE:
13    ais rmo BUCKET[/OBJECT_NAME_or_TEMPLATE] [BUCKET[/OBJECT_NAME_or_TEMPLATE] ...] [command options]
14 
15 OPTIONS:
16    --all                  Remove all objects (use with extreme caution!)
17    --list value           Comma-separated list of object or file names, e.g.:
18                           --list 'o1,o2,o3'
19                           --list "abc/1.tar, abc/1.cls, abc/1.jpeg"
20                           or, when listing files and/or directories:
21                           --list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
22    --non-recursive, --nr  Non-recursive operation, e.g.:
23                           - 'ais ls gs://bucket/prefix --nr'   - list objects and/or virtual subdirectories with names starting with the specified prefix;
24                           - 'ais ls gs://bucket/prefix/ --nr'  - list contained objects and/or immediately nested virtual subdirectories _without_ recursing into the latter;
25                           - 'ais prefetch s3://bck/abcd --nr'  - prefetch a single named object (see 'ais prefetch --help' for details);
26                           - 'ais rmo gs://bucket/prefix --nr'  - remove a single object with the specified name (see 'ais rmo --help' for details)
27    --non-verbose, --nv    Non-verbose (quiet) output, minimized reporting, fewer warnings
28    --prefix value         Select virtual directories or objects with names starting with the specified prefix, e.g.:
29                           '--prefix a/b/c'   - matches names 'a/b/c/d', 'a/b/cdef', and similar;
30                           '--prefix a/b/c/'  - only matches objects from the virtual directory a/b/c/
31    --progress             Show progress bar(s) and progress of execution in real time
32    --refresh value        Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
33                           valid time units: ns, us (or µs), ms, s (default), m, h
34    --skip-lookup          Do not execute HEAD(bucket) request to lookup remote bucket and its properties; possible usage scenarios include:
35                            1) adding remote bucket to aistore without first checking the bucket's accessibility
36                               (e.g., to configure the bucket's aistore properties with alternative security profile and/or endpoint)
37                            2) listing public-access Cloud buckets where certain operations (e.g., 'HEAD(bucket)') may be disallowed
38    --template value       Template to match object or file names; may contain prefix (that could be empty) with zero or more ranges
39                           (with optional steps and gaps), e.g.:
40                           --template "" # (an empty or '*' template matches everything)
41                           --template 'dir/subdir/'
42                           --template 'shard-{1000..9999}.tar'
43                           --template "prefix-{0010..0013..2}-gap-{1..2}-suffix"
44                           and similarly, when specifying files and directories:
45                           --template '/home/dir/subdir/'
46                           --template "/abc/prefix-{0010..9999..2}-suffix"
47    --timeout value        Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
48                           valid time units: ns, us (or µs), ms, s (default), m, h
49    --verbose, -v          Verbose output
50    --wait                 Wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
51    --yes, -y              Assume 'yes' to all questions
52    --help, -h             Show help

For multi-object delete operation, see also Operations on Lists and Ranges (and entire buckets) below.

Disambiguating multi-object operation

Let’s say, in its initial state the bucket consists of:

1 $ ais ls s3://mybucket
2 NAME                                     SIZE            CACHED
3 README.md                                16.26KiB        no
4 aaa                                      16.26KiB        yes
5 aaa/bbb/111                              16.26KiB        no
6 aaa/bbb/ccc/README.md                    5.09KiB         no
7 ...

Notice that aaa here is both an object and a virtual directory.

That’s why:

1 $ ais rmo s3://mybucket/aaa
2 Error: part of the URI "aaa" can be interpreted as an object name and/or mutli-object matching prefix
3 (Tip:  to disambiguate, use either '--non-recursive' or '--prefix')

And so, as per the Tip (above), we can go ahead and disambiguate one way or another, e.g.:

1 $ ais rmo s3://mybucket/aaa --nr
2 deleted "aaa" from s3://mybucket
3 
4 $ ais ls s3://mybucket
5 NAME                                     SIZE            CACHED
6 README.md                                16.26KiB        no
7 aaa/bbb/111                              16.26KiB        no
8 aaa/bbb/ccc/README.md                    5.09KiB         no
9 ...

Delete a single object

Delete object myobj.tgz from bucket mybucket.

1 $ ais object rm ais://mybucket/myobj.tgz
2 myobj.tgz deleted from ais://mybucket bucket

Delete multiple space-separated objects

Delete objects (obj1, obj2) from buckets (aisbck, cloudbck) respectively.

1 $ ais object rm ais://aisbck/obj1.tgz aws://cloudbck/obj2.tgz
2 obj1.tgz deleted from ais://aisbck bucket
3 obj2.tgz deleted from aws://cloudbck bucket

NOTE: for each space-separated object name CLI sends a separate request.
For multi-object delete that operates on a --list or --template, please see: Operations on Lists and Ranges (and entire buckets) below.

Evict one remote bucket, multiple remote buckets, or selected objects in a given remote bucket or buckets

Some of the supported functionality can be quickly demonstrated with the following examples:

CLI: Three Ways to Evict Remote Bucket

1 $ ais evict --help
2 NAME:
3    ais evict - (alias for "bucket evict") Evict one remote bucket, multiple remote buckets, or
4      selected objects in a given remote bucket or buckets,
5      e.g.:
6      - evict gs://abc                                          - evict entire bucket from aistore: remove all "cached" gs://abc objects _and_ bucket metadata;
7      - evict gs://abc --keep-md                                - same as above but keep bucket metadata;
8      - evict gs:                                               - evict all GCP buckets from the cluster;
9      - evict gs://abc --prefix images/                         - evict all gs://abc objects from the virtual subdirectory "images";
10      - evict gs://abc/images/                                  - same as above;
11      - evict gs://abc/images/ --nr                             - same as above, but do not recurse into virtual subdirs;
12      - evict gs://abc --template images/                       - same as above;
13      - evict gs://abc --template "shard-{0000..9999}.tar.lz4"  - evict the matching range (prefix + brace expansion);
14      - evict "gs://abc/shard-{0000..9999}.tar.lz4"             - same as above (notice BUCKET/TEMPLATE argument in quotes)
15 
16 USAGE:
17    ais evict BUCKET[/OBJECT_NAME_or_TEMPLATE] [BUCKET[/OBJECT_NAME_or_TEMPLATE] ...] [command options]
18 
19 OPTIONS:
20    dry-run           Preview the results without really running the action
21    keep-md,k         Keep bucket metadata
22    list              Comma-separated list of object or file names, e.g.:
23                      --list 'o1,o2,o3'
24                      --list "abc/1.tar, abc/1.cls, abc/1.jpeg"
25                      or, when listing files and/or directories:
26                      --list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
27    non-recursive,nr  Non-recursive operation, e.g.:
28                      - 'ais ls gs://bucket/prefix --nr'   - list objects and/or virtual subdirectories with names starting with the specified prefix;
29                      - 'ais ls gs://bucket/prefix/ --nr'  - list contained objects and/or immediately nested virtual subdirectories _without_ recursing into the latter;
30                      - 'ais prefetch s3://bck/abcd --nr'  - prefetch a single named object (see 'ais prefetch --help' for details);
31                      - 'ais rmo gs://bucket/prefix --nr'  - remove a single object with the specified name (see 'ais rmo --help' for details)
32    non-verbose,nv    Non-verbose (quiet) output, minimized reporting, fewer warnings
33    prefix            Select virtual directories or objects with names starting with the specified prefix, e.g.:
34                      '--prefix a/b/c'   - matches names 'a/b/c/d', 'a/b/cdef', and similar;
35                      '--prefix a/b/c/'  - only matches objects from the virtual directory a/b/c/
36    progress          Show progress bar(s) and progress of execution in real time
37    refresh           Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
38                      valid time units: ns, us (or µs), ms, s (default), m, h
39    skip-lookup       Do not execute HEAD(bucket) request to lookup remote bucket and its properties; possible usage scenarios include:
40                       1) adding remote bucket to aistore without first checking the bucket's accessibility
41                          (e.g., to configure the bucket's aistore properties with alternative security profile and/or endpoint)
42                       2) listing public-access Cloud buckets where certain operations (e.g., 'HEAD(bucket)') may be disallowed
43    template          Template to match object or file names; may contain prefix (that could be empty) with zero or more ranges
44                      (with optional steps and gaps), e.g.:
45                      --template "" # (an empty or '*' template matches everything)
46                      --template 'dir/subdir/'
47                      --template 'shard-{1000..9999}.tar'
48                      --template "prefix-{0010..0013..2}-gap-{1..2}-suffix"
49                      and similarly, when specifying files and directories:
50                      --template '/home/dir/subdir/'
51                      --template "/abc/prefix-{0010..9999..2}-suffix"
52    timeout           Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
53                      valid time units: ns, us (or µs), ms, s (default), m, h
54    verbose,v         Verbose output
55    wait              Wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
56    help, h           Show help

Evict object(s) from a bucket that has remote backend.

NOTE: for each space-separated object name CLI sends a separate request.
For multi-object eviction that operates on a --list or --template, please see: Operations on Lists and Ranges (and entire buckets) below.
Similar to delete, prefetch and copy operations, evict also supports embedded prefix - see disambiguating multi-object operation

Evict a single object

Put file.txt object to cloudbucket bucket and evict it locally.

1 $ ais put file.txt aws://cloudbucket/file.txt
2 PUT file.txt into bucket aws://cloudbucket
3 
4 $ ais bucket summary aws://cloudbucket --cached # show only cloudbucket objects present in the AIS cluster
5 NAME	           OBJECTS	 SIZE    USED %
6 aws://cloudbucket  1             702B    0%
7 
8 $ ais bucket evict aws://cloudbucket/file.txt
9 file.txt evicted from aws://cloudbucket bucket
10 
11 $ ais bucket summary aws://cloudbucket --cached
12 NAME	           OBJECTS	 SIZE    USED %
13 aws://cloudbucket  0             0B      0%

Evict a range of objects

1 $ ais bucket evict aws://cloudbucket --template "shard-{900..999}.tar"

Move object

ais object mv BUCKET/OBJECT_NAME NEW_OBJECT_NAME

Move (rename) an object within an ais bucket. Moving objects from one bucket to another bucket is not supported. If the NEW_OBJECT_NAME already exists, it will be overwritten without confirmation.

Concat objects

ais object concat DIRNAME|FILENAME [DIRNAME|FILENAME...] BUCKET/OBJECT_NAME

Create an object in a bucket by concatenating the provided files in the order of the arguments provided. If an object of the same name exists, the object will be overwritten without confirmation.

If a directory is provided, files within the directory are sent in lexical order of filename to the cluster for concatenation. Recursive iteration through directories and wildcards is supported in the same way as the PUT operation.

Options

1 $ ais object concat --help
2 
3 NAME:
4    ais object concat - Append a file, a directory, or multiple files and/or directories
5    as a new BUCKET/OBJECT_NAME if doesn't exists, and to an existing BUCKET/OBJECT_NAME otherwise, e.g.:
6    $ ais object concat docs ais://nnn/all-docs ### concatenate all files from docs/ directory.
7 
8 USAGE:
9    ais object concat FILE|DIRECTORY[/PATTERN] [ FILE|DIRECTORY[/PATTERN] ...] BUCKET/OBJECT_NAME [command options]
10 
11 OPTIONS:
12    --progress       Show progress bar(s) and progress of execution in real time
13    --recursive, -r  Recursive operation
14    --units value    Show statistics and/or parse command-line specified sizes using one of the following units of measurement:
15                     iec - IEC format, e.g.: KiB, MiB, GiB (default)
16                     si  - SI (metric) format, e.g.: KB, MB, GB
17                     raw - do not convert to (or from) human-readable format
18    --help, -h       Show help

Concat two files

In two separate requests sends file1.txt and dir/file2.txt to the cluster, concatenates the files keeping the order and saves them as obj in bucket mybucket.

1 $ ais object concat file1.txt dir/file2.txt ais://mybucket/obj

Concat with progress bar

Same as above, but additionally shows progress bar of sending the files to the cluster.

1 $ ais object concat file1.txt dir/file2.txt ais://mybucket/obj --progress

Concat files from directories

Creates obj in bucket mybucket which is concatenation of sorted files from dirB with sorted files from dirA.

1 $ ais object concat dirB dirA ais://mybucket/obj

Set custom properties

Generally, AIS objects have two kinds of properties: system and, optionally, custom (user-defined). Unlike the system-maintained properties, such as checksum and the number of copies (or EC parity slices, etc.), custom properties may have arbitrary user-defined names and values.

Custom properties are not impacted by object updates (PUTs) — a new version of an object simply inherits custom properties of the previous version as is with no changes.

The command’s syntax is similar to the one used to assign bucket properties

ais object set-custom BUCKET/OBJECT_NAME JSON_SPECIFICATION|KEY=VALUE [KEY=VALUE...], [command options]

for example:

1 $ ais put README.md ais://abc
2 $ ais object set-custom ais://abc/README.md mykey1=value1 mykey2=value2
3 
4 # or, the same using JSON formatting:
5 $ ais object set-custom ais://abc/README.md '{"mykey1":"value1", "mykey2":"value2"}'

To show the results:

1 $ ais show object ais://abc/README.md --props=all
2 PROPERTY         VALUE
3 atime            30 Jun 21 09:43 PDT
4 cached           yes
5 checksum         47904b6991a92ca9
6 copies           1
7 custom           mykey1=value1, mykey2=value2
8 ec               -
9 name             ais://abc/README.md
10 size             13.13KiB
11 version          1

Note the flag --props=all used to show all object’s properties including the custom ones, if available.

Operations on Lists and Ranges (and entire buckets)

Generally, multi-object operations are supported in 2 different ways:

specifying source directory in the command line - see e.g. Promote files and directories and Concat objects;
via --list or --template options, whereby the latter supports Bash expansion syntax and can also contain prefix, such as a virtual parent directory, etc.)

This section documents and exemplifies AIS CLI operating on multiple (source) objects that you can specify either explicitly or implicitly using the --list or --template flags.

The number of objects “involved” in a single operation does not have any designed-in limitations: all AIS targets work on a given multi-object operation simultaneously and in parallel.

See also: List/Range Operations.

Prefetch objects

This is ais start prefetch or, same, ais prefetch command:

1 $ ais prefetch --help
2 NAME:
3    ais prefetch - (alias for "object prefetch") Prefetch one remote bucket, multiple remote buckets, or
4    selected objects in a given remote bucket or buckets, e.g.:
5      - 'prefetch gs://abc'                                          - prefetch entire bucket (all gs://abc objects that are _not_ in-cluster);
6      - 'prefetch gs://abc --num-workers 32'                         - same as above with 32 concurrent (prefetching) workers;
7      - 'prefetch gs:'                                               - prefetch all visible/accessible GCP buckets;
8      - 'prefetch gs: --num-workers=48'                              - same as above employing 48 workers;
9      - 'prefetch gs://abc --prefix images/'                         - prefetch all objects from the virtual subdirectory "images";
10      - 'prefetch gs://abc --prefix images/ --nr'                    - prefetch only immediate contents of "images/" (non-recursive);
11      - 'prefetch gs://abc --template images/'                       - same as above;
12      - 'prefetch gs://abc/images/'                                  - same as above;
13      - 'prefetch gs://abc --template "shard-{0000..9999}.tar.lz4"'  - prefetch the matching range (prefix + brace expansion);
14      - 'prefetch "gs://abc/shard-{0000..9999}.tar.lz4"'             - same as above (notice double quotes)
15 
16 USAGE:
17    ais prefetch BUCKET[/OBJECT_NAME_or_TEMPLATE] [BUCKET[/OBJECT_NAME_or_TEMPLATE] ...] [command options]
18 
19 OPTIONS:
20    --blob-threshold value  Utilize built-in blob-downloader for remote objects greater than the specified (threshold) size
21                            in IEC or SI units, or "raw" bytes (e.g.: 4mb, 1MiB, 1048576, 128k; see '--units')
22    --dry-run               Preview the results without really running the action
23    --latest                Check in-cluster metadata and, possibly, GET, download, prefetch, or otherwise copy the latest object version
24                            from the associated remote bucket;
25                            the option provides operation-level control over object versioning (and version synchronization)
26                            without the need to change the corresponding bucket configuration: 'versioning.validate_warm_get';
27                            see also:
28                              - 'ais show bucket BUCKET versioning'
29                              - 'ais bucket props set BUCKET versioning'
30                              - 'ais ls --check-versions'
31                            supported commands include:
32                              - 'ais cp', 'ais prefetch', 'ais get'
33    --list value            Comma-separated list of object or file names, e.g.:
34                            --list 'o1,o2,o3'
35                            --list "abc/1.tar, abc/1.cls, abc/1.jpeg"
36                            or, when listing files and/or directories:
37                            --list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
38    --non-recursive, --nr   Non-recursive operation, e.g.:
39                            - 'ais ls gs://bucket/prefix --nr'   - list objects and/or virtual subdirectories with names starting with the specified prefix;
40                            - 'ais ls gs://bucket/prefix/ --nr'  - list contained objects and/or immediately nested virtual subdirectories _without_ recursing into the latter;
41                            - 'ais prefetch s3://bck/abcd --nr'  - prefetch a single named object (see 'ais prefetch --help' for details);
42                            - 'ais rmo gs://bucket/prefix --nr'  - remove a single object with the specified name (see 'ais rmo --help' for details)
43    --num-workers value     Number of concurrent workers (readers); defaults to a number of target mountpaths if omitted or zero;
44                            use (-1) to indicate single-threaded serial execution (ie., no workers);
45                            any positive value will be adjusted _not_ to exceed the number of target CPUs (default: 0)
46    --prefix value          Select virtual directories or objects with names starting with the specified prefix, e.g.:
47                            '--prefix a/b/c'   - matches names 'a/b/c/d', 'a/b/cdef', and similar;
48                            '--prefix a/b/c/'  - only matches objects from the virtual directory a/b/c/
49    --progress              Show progress bar(s) and progress of execution in real time
50    --refresh value         Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
51                            valid time units: ns, us (or µs), ms, s (default), m, h
52    --skip-lookup           Do not execute HEAD(bucket) request to lookup remote bucket and its properties; possible usage scenarios include:
53                             1) adding remote bucket to aistore without first checking the bucket's accessibility
54                                (e.g., to configure the bucket's aistore properties with alternative security profile and/or endpoint)
55                             2) listing public-access Cloud buckets where certain operations (e.g., 'HEAD(bucket)') may be disallowed
56    --template value        Template to match object or file names; may contain prefix (that could be empty) with zero or more ranges
57                            (with optional steps and gaps), e.g.:
58                            --template "" # (an empty or '*' template matches everything)
59                            --template 'dir/subdir/'
60                            --template 'shard-{1000..9999}.tar'
61                            --template "prefix-{0010..0013..2}-gap-{1..2}-suffix"
62                            and similarly, when specifying files and directories:
63                            --template '/home/dir/subdir/'
64                            --template "/abc/prefix-{0010..9999..2}-suffix"
65    --timeout value         Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
66                            valid time units: ns, us (or µs), ms, s (default), m, h
67    --wait                  Wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
68    --yes, -y               Assume 'yes' to all questions
69    --help, -h              Show help

Note usage examples above. You can always run --help option to see the most recently updated inline help.

Example prefetching objects

This example demonstrates how to prefetch objects from a remote bucket, and how to monitor the progress of the operation.

Checking cached objects

First, let’s check which objects are currently stored in-cluster (if any):

1 $ ais ls s3://cloud-bucket --cached
2 NAME     SIZE
3 1000052  1.00MiB
4 10000a2  1.00MiB
5 10000b4  1.00MiB
6 10000bd  1.00MiB
7 ...

Evicting cached objects

To remove all in-cluster content while preserving the bucket’s metadata:

The terms in-cluster and cached are used interchangeably throughout the entire documentation and CLI.

1 $ ais evict s3://cloud-bucket --keep-md
2 Evicted s3://cloud-bucket contents from aistore: the bucket is now empty
3 
4 $ ais ls s3://cloud-bucket --cached
5 NAME     SIZE

Prefetching objects

To prefetch objects with a specific prefix from a cloud bucket:

1 $ ais prefetch s3://cloud-bucket --prefix 10 --num-workers 16
2 
3 prefetch-objects[MV4ex8u6h]: prefetch "10" from s3://cloud-bucket. To monitor the progress, run 'ais show job MV4ex8u6h'

The prefix in the example is “10”

Monitoring progress

You can monitor the progress of the prefetch operation using the ais show job prefetch command. Add the --refresh flag followed by a time in seconds to get automatic updates:

1 $ ais show job prefetch
2 
3 prefetch-objects[MV4ex8u6h] (ctl: prefix:10, workers: 16, parallelism: w[16] chan-full[0,6])
4 NODE             ID              KIND                    BUCKET          OBJECTS         BYTES           START           END     STATE
5 KactABCD         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 5               5.00MiB         18:28:55        -       Running
6 XXytEFGH         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 4               4.00MiB         18:28:55        -       Running
7 YMjtIJKL         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 5               5.00MiB         18:28:55        -       Running
8 oJXtMNOP         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 6               6.00MiB         18:28:55        -       Running
9 vWrtQRST         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 5               5.00MiB         18:28:55        -       Running
10 ybTtUVWX         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 5               5.00MiB         18:28:55        -       Running
11                                 Total:                                  30              30.00MiB ✓

The output shows statistics for each node in the AIStore cluster:

NODE: The name of the node
ID: The job ID
KIND: The type of operation
BUCKET: Source bucket
OBJECTS: Number of objects processed
BYTES: Amount of data prefetched
START: Job start time
END: Job end time (empty if job is still running)
STATE: Current job state

The output also includes a “Total” row at the bottom that provides cluster-wide aggregated values for the number of objects prefetched and bytes transferred. The checkmark (✓) indicates that all nodes are reporting byte statistics.

You can see the progress over time with automatic refresh:

1 $ ais show job prefetch --refresh 10
2 
3 prefetch-objects[MV4ex8u6h] (ctl: prefix:10, workers: 16, parallelism: w[16] chan-full[8,32])
4 NODE             ID              KIND                    BUCKET          OBJECTS         BYTES           START           END     STATE
5 KactABCD         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 27              27.00MiB        18:28:55        -       Running
6 XXytEFGH         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 23              23.00MiB        18:28:55        -       Running
7 YMjtIJKL         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 41              41.00MiB        18:28:55        -       Running
8 oJXtMNOP         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 34              34.00MiB        18:28:55        -       Running
9 vWrtQRST         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 23              23.00MiB        18:28:55        -       Running
10 ybTtUVWX         MV4ex8u6h       prefetch-listrange      s3://cloud-bucket 31              31.00MiB        18:28:55        -       Running
11                                 Total:                                  179             179.00MiB ✓

Stopping jobs

To stop all in-progress jobs:

1 $ ais stop --all

This will stop all running jobs. To stop a specific job, use ais stop job JOB_ID.

Example: prefetch using prefix

Initially:

1 $ ais ls s3://abc --all --limit 10
2 NAME     SIZE            CACHED  STATUS
3 10000a2  10.00MiB        no      n/a
4 10000b4  10.00MiB        no      n/a
5 10000bd  10.00MiB        no      n/a
6 10000d6  10.00MiB        no      n/a
7 10000ea  10.00MiB        no      n/a
8 10001a2  10.00MiB        no      n/a
9 10001b4  10.00MiB        no      n/a
10 10001bd  10.00MiB        no      n/a
11 10001d6  10.00MiB        no      n/a
12 10001ea  10.00MiB        no      n/a

Now, let’s use --prefix option to - in this case - fetch a single object:

1 $ ais prefetch s3://abc --prefix 10000a2
2 prefetch-objects[E0e5mq9Kav]: prefetch "10000a2" from s3://abc. To monitor the progress, run 'ais show job E0e5mq9Kav'
3 
4 $ ais ls s3://abc --all --limit 10
5 NAME     SIZE            CACHED  STATUS
6 10000a2  10.00MiB        yes     ok     ### <<<<< in cluster
7 10000b4  10.00MiB        no      n/a
8 10000bd  10.00MiB        no      n/a
9 10000d6  10.00MiB        no      n/a
10 10000ea  10.00MiB        no      n/a
11 10001a2  10.00MiB        no      n/a
12 10001b4  10.00MiB        no      n/a
13 10001bd  10.00MiB        no      n/a
14 10001d6  10.00MiB        no      n/a
15 10001ea  10.00MiB        no      n/a

Example: prefetch using template

Since --template can optionally contain prefix and zero or more ranges, we could execute the above example as follows:

1 $ ais prefetch s3://abc --template 10000a2

This, in fact, would produce the same result (see previous section).

But of course, “templated” match can also specify an actual range, for example:

1 $ ais ls gs://nnn --all --limit 5
2 NAME     SIZE            CACHED  STATUS
3 shard-001  1.00MiB       no      n/a
4 shard-002  1.00MiB       no      n/a
5 shard-003  1.00MiB       no      n/a
6 shard-004  1.00MiB       no      n/a
7 shard-005  1.00MiB       no      n/a
8 
9 $ ais prefetch gs://nnn --template --template "shard-{001..003}"
10 
11 $ ais ls gs://nnn --all --limit 5
12 NAME     SIZE            CACHED  STATUS
13 shard-001  1.00MiB       yes     ok
14 shard-002  1.00MiB       yes     ok
15 shard-003  1.00MiB       yes     ok
16 shard-004  1.00MiB       no      n/a
17 shard-005  1.00MiB       no      n/a

Example: prefetch a list of objects

NOTE: make sure to use double or single quotations to specify the list, as shown below.

1 # Prefetch o1, o2, and o3 from AWS bucket `cloudbucket`:
2 $ ais prefetch aws://cloudbucket --list 'o1,o2,o3'

Example: prefetch a range of objects

1 # Prefetch from AWS bucket `cloudbucket` all objects in the specified range.
2 # NOTE: make sure to use double or single quotations to specify the template (aka "range")
3 
4 $ ais prefetch aws://cloudbucket --template "shard-{001..999}.tar"

Delete multiple objects

ais object rm BUCKET[/OBJECT_NAME_or_TEMPLATE] [BUCKET[/OBJECT_NAME_or_TEMPLATE] ...] [command options]

Delete an object or list or range of objects from a bucket.

Alias: ais rmo.

Options

1 $ ais object rm --help
2 NAME:
3    ais object rm - Remove object or selected objects from the specified bucket, or buckets - e.g.:
4      - 'rm ais://nnn --all'                                   - remove all objects from the bucket ais://nnn;
5      - 'rm s3://abc' --all                                    - remove all objects including those that are not _present_ in the cluster;
6      - 'rm gs://abc --prefix images/'                         - remove all objects from the virtual subdirectory "images";
7      - 'rm gs://abc/images/'                                  - same as above;
8      - 'rm gs://abc/images/ --nr'                             - same as above, but do not recurse into virtual subdirs;
9      - 'rm gs://abc --template images/'                       - same as above;
10      - 'rm gs://abc --template "shard-{0000..9999}.tar.lz4"'  - remove the matching range (prefix + brace expansion);
11      - 'rm "gs://abc/shard-{0000..9999}.tar.lz4"'             - same as above (notice BUCKET/TEMPLATE argument in quotes)
12 
13 USAGE:
14    ais object rm BUCKET[/OBJECT_NAME_or_TEMPLATE] [BUCKET[/OBJECT_NAME_or_TEMPLATE] ...] [command options]
15 
16 OPTIONS:
17    --all                  Remove all objects (use with extreme caution!)
18    --list value           Comma-separated list of object or file names, e.g.:
19                           --list 'o1,o2,o3'
20                           --list "abc/1.tar, abc/1.cls, abc/1.jpeg"
21                           or, when listing files and/or directories:
22                           --list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
23    --non-recursive, --nr  Non-recursive operation, e.g.:
24                           - 'ais ls gs://bucket/prefix --nr'   - list objects and/or virtual subdirectories with names starting with the specified prefix;
25                           - 'ais ls gs://bucket/prefix/ --nr'  - list contained objects and/or immediately nested virtual subdirectories _without_ recursing into the latter;
26                           - 'ais prefetch s3://bck/abcd --nr'  - prefetch a single named object (see 'ais prefetch --help' for details);
27                           - 'ais rmo gs://bucket/prefix --nr'  - remove a single object with the specified name (see 'ais rmo --help' for details)
28    --non-verbose, --nv    Non-verbose (quiet) output, minimized reporting, fewer warnings
29    --prefix value         Select virtual directories or objects with names starting with the specified prefix, e.g.:
30                           '--prefix a/b/c'   - matches names 'a/b/c/d', 'a/b/cdef', and similar;
31                           '--prefix a/b/c/'  - only matches objects from the virtual directory a/b/c/
32    --progress             Show progress bar(s) and progress of execution in real time
33    --refresh value        Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
34                           valid time units: ns, us (or µs), ms, s (default), m, h
35    --skip-lookup          Do not execute HEAD(bucket) request to lookup remote bucket and its properties; possible usage scenarios include:
36                            1) adding remote bucket to aistore without first checking the bucket's accessibility
37                               (e.g., to configure the bucket's aistore properties with alternative security profile and/or endpoint)
38                            2) listing public-access Cloud buckets where certain operations (e.g., 'HEAD(bucket)') may be disallowed
39    --template value       Template to match object or file names; may contain prefix (that could be empty) with zero or more ranges
40                           (with optional steps and gaps), e.g.:
41                           --template "" # (an empty or '*' template matches everything)
42                           --template 'dir/subdir/'
43                           --template 'shard-{1000..9999}.tar'
44                           --template "prefix-{0010..0013..2}-gap-{1..2}-suffix"
45                           and similarly, when specifying files and directories:
46                           --template '/home/dir/subdir/'
47                           --template "/abc/prefix-{0010..9999..2}-suffix"
48    --timeout value        Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
49                           valid time units: ns, us (or µs), ms, s (default), m, h
50    --verbose, -v          Verbose output
51    --wait                 Wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
52    --yes, -y              Assume 'yes' to all questions
53    --help, -h             Show help

Delete a list of objects

Delete a list of objects (obj1, obj2, obj3) from bucket mybucket.

NOTE: when specifying a comma-delimited --list option, make sure to use double or single quotations as shown below.

1 $ ais object rm ais://mybucket --list "obj1, obj2, obj3"
2 [obj1 obj2] removed from ais://mybucket bucket

Delete a range of objects

1 # Delete from bucket `mybucket` all objects in the range `001-003` with prefix `test-`.
2 # NOTE: when specifying template (aka "range") make sure to use double or single quotation marks.
3 
4 $ ais object rm ais://mybucket --template "test-{001..003}"
5 removed files in the range 'test-{001..003}' from ais://mybucket bucket

And one other example (that also includes generating .tar shards):

1 $ ais archive gen-shards "ais://dsort-testing/shard-{001..999}.tar" --fcount 256
2 Shards created: 999/999 [==============================================================] 100 %
3 
4 # NOTE: make sure to use double or single quotations to specify the template (aka "range")
5 $ ais object rm ais://dsort-testing --template 'shard-{900..999}.tar'
6 removed from ais://dsort-testing objects in the range "shard-{900..999}.tar", use 'ais job show xaction EH291ljOy' to monitor the progress

Evict multiple objects

ais evict BUCKET[/OBJECT_NAME_or_TEMPLATE] [BUCKET[/OBJECT_NAME_or_TEMPLATE] ...] [command options]

Command ais evict is a shorter version of ais bucket evict.

Options

Here’s inline help, and specifically notice the multi-object options: --template, --list, and --prefix:

1 $ ais evict --help
2 NAME:
3    ais evict - (alias for "bucket evict") Evict one remote bucket, multiple remote buckets, or
4      selected objects in a given remote bucket or buckets,
5      e.g.:
6      - evict gs://abc                                          - evict entire bucket from aistore: remove all "cached" gs://abc objects _and_ bucket metadata;
7      - evict gs://abc --keep-md                                - same as above but keep bucket metadata;
8      - evict gs:                                               - evict all GCP buckets from the cluster;
9      - evict gs://abc --prefix images/                         - evict all gs://abc objects from the virtual subdirectory "images";
10      - evict gs://abc/images/                                  - same as above;
11      - evict gs://abc/images/ --nr                             - same as above, but do not recurse into virtual subdirs;
12      - evict gs://abc --template images/                       - same as above;
13      - evict gs://abc --template "shard-{0000..9999}.tar.lz4"  - evict the matching range (prefix + brace expansion);
14      - evict "gs://abc/shard-{0000..9999}.tar.lz4"             - same as above (notice BUCKET/TEMPLATE argument in quotes)
15 
16 USAGE:
17    ais evict BUCKET[/OBJECT_NAME_or_TEMPLATE] [BUCKET[/OBJECT_NAME_or_TEMPLATE] ...] [command options]
18 
19 OPTIONS:
20    dry-run           Preview the results without really running the action
21    keep-md           Keep bucket metadata
22    list              Comma-separated list of object or file names, e.g.:
23                      --list 'o1,o2,o3'
24                      --list "abc/1.tar, abc/1.cls, abc/1.jpeg"
25                      or, when listing files and/or directories:
26                      --list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
27    non-recursive,nr  Non-recursive operation, e.g.:
28                      - 'ais ls gs://bucket/prefix --nr'   - list objects and/or virtual subdirectories with names starting with the specified prefix;
29                      - 'ais ls gs://bucket/prefix/ --nr'  - list contained objects and/or immediately nested virtual subdirectories _without_ recursing into the latter;
30                      - 'ais prefetch s3://bck/abcd --nr'  - prefetch a single named object (see 'ais prefetch --help' for details);
31                      - 'ais rmo gs://bucket/prefix --nr'  - remove a single object with the specified name (see 'ais rmo --help' for details)
32    non-verbose,nv    Non-verbose (quiet) output, minimized reporting, fewer warnings
33    prefix            Select virtual directories or objects with names starting with the specified prefix, e.g.:
34                      '--prefix a/b/c'   - matches names 'a/b/c/d', 'a/b/cdef', and similar;
35                      '--prefix a/b/c/'  - only matches objects from the virtual directory a/b/c/
36    progress          Show progress bar(s) and progress of execution in real time
37    refresh           Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
38                      valid time units: ns, us (or µs), ms, s (default), m, h
39    skip-lookup       Do not execute HEAD(bucket) request to lookup remote bucket and its properties; possible usage scenarios include:
40                       1) adding remote bucket to aistore without first checking the bucket's accessibility
41                          (e.g., to configure the bucket's aistore properties with alternative security profile and/or endpoint)
42                       2) listing public-access Cloud buckets where certain operations (e.g., 'HEAD(bucket)') may be disallowed
43    template          Template to match object or file names; may contain prefix (that could be empty) with zero or more ranges
44                      (with optional steps and gaps), e.g.:
45                      --template "" # (an empty or '*' template matches everything)
46                      --template 'dir/subdir/'
47                      --template 'shard-{1000..9999}.tar'
48                      --template "prefix-{0010..0013..2}-gap-{1..2}-suffix"
49                      and similarly, when specifying files and directories:
50                      --template '/home/dir/subdir/'
51                      --template "/abc/prefix-{0010..9999..2}-suffix"
52    timeout           Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
53                      valid time units: ns, us (or µs), ms, s (default), m, h
54    verbose,v         Verbose output
55    wait              Wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
56    help, h           Show help

Note usage examples above. You can always run --help option to see the most recently updated inline help.

Evict a range of objects

1 $ ais bucket evict aws://cloudbucket --template "shard-{900..999}.tar"

Table of Contents

GET object

Save object to local file

Save object to local file with implied file name

Get object and print it to standard output

Check if object is cached

Read range

Example: read-range multiple objects

Multipart Download

GET multiple objects

GET archived content

Example: extracting one file using its fully-qualified name::

Example: extract all files from all shards with a given prefix

Example: use ‘—prefix’ that crosses shard boundary

Print object content

Options

Print content of object

Read range

Show object properties

Show default object properties

Show all object properties

Show selected object properties

PUT object

Inline help

Object names

Put with client-side checksumming

First PUT:

Second PUT with no changes at the source:

Adding one file to the source:

Put single file

Put single file with checksum

Put single file with implicitly defined name

Put content from STDIN

Put directory

Put multiple files with prefix added to destination object names

Example 1.

Example 2.

Example 3.

Example 4.

Put multiple files into virtual directory, track progress

Put pattern-matching files from directory

Put a range of files

Example 1.

Example 2. PUT a range of files into a virtual directory

Example 3.

Put a list of files

Example 1. Notice the double quotes (single quotes can be used as well)

Example 2.

Example 3. PUT a list into virtual directory

Dry-Run option

Example 1

Example 2

Put multiple directories using Bash range notation

Put multiple directories using filename-matching pattern (wildcard)

Put multiple directories with the --skip-vc option

Tips for Copying Files from Lustre (NFS)

Disclaimer

Tips

Example 1

Example 2

Example 3

Example 4

Promote files and directories

Options

Destination naming

Promote a single file

Promote file while specifying custom (resulting) name

Promote a directory

Promote directory with custom prefix

Promote invalid path

Multipart upload

Options

Create multipart upload

Create a multipart upload session

Upload parts

Upload parts sequentially

Upload parts in parallel

Complete multipart upload

Complete upload with all parts

Complete upload using flags

Put multiple directories with the `--skip-vc` option