Cli

Blob Downloader

View as Markdown

AIS comes with built-in blob downloading facility that employs multiple concurrent readers to speed-up reading very large remote objects.

In terms of its from/to operation, blob downloading can be compared to prefetching.

More precisely, the list of “comparables” includes:

In all these cases, destination of the remote content is aistore cluster (and not the client requesting the operation).

In CLI, ais blob-download will run a job, or multiple jobs, to download user-specified remote blob(s). Command line options and examples follow below.

Options

1$ ais blob-download --help
2
3NAME:
4   ais blob-download - (alias for "job start blob-download") Download a large object or multiple objects from remote storage, e.g.:
5     - 'blob-download s3://ab/largefile --chunk-size=2mb --progress'       - download one blob at a given chunk size
6     - 'blob-download s3://ab --list "f1, f2" --num-workers=4 --progress'  - run 4 concurrent readers to download 2 (listed) blobs
7   When _not_ using '--progress' option, run 'ais show job' to monitor.
8
9USAGE:
10   ais blob-download BUCKET/OBJECT_NAME [command options]
11
12OPTIONS:
13   --chunk-size value   Chunk size in IEC or SI units, or "raw" bytes (e.g.: 4mb, 1MiB, 1048576, 128k; see '--units')
14   --latest             Check in-cluster metadata and, possibly, GET, download, prefetch, or otherwise copy the latest object version
15                        from the associated remote bucket;
16                        the option provides operation-level control over object versioning (and version synchronization)
17                        without the need to change the corresponding bucket configuration: 'versioning.validate_warm_get';
18                        see also:
19                          - 'ais show bucket BUCKET versioning'
20                          - 'ais bucket props set BUCKET versioning'
21                          - 'ais ls --check-versions'
22                        supported commands include:
23                          - 'ais cp', 'ais prefetch', 'ais get'
24   --list value         Comma-separated list of object or file names, e.g.:
25                        --list 'o1,o2,o3'
26                        --list "abc/1.tar, abc/1.cls, abc/1.jpeg"
27                        or, when listing files and/or directories:
28                        --list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
29   --non-verbose, --nv  Non-verbose (quiet) output, minimized reporting, fewer warnings
30   --num-workers value  Number of concurrent blob-downloading workers (readers); system default when omitted or zero (default: 0)
31   --progress           Show progress bar(s) and progress of execution in real time
32   --refresh value      Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
33                        valid time units: ns, us (or µs), ms, s (default), m, h
34   --timeout value      Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
35                        valid time units: ns, us (or µs), ms, s (default), m, h
36   --wait               Wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
37   --help, -h           Show help

Usage example

1$ ais blob-download s3://abc --list "file-2gb, file-100mb" --chunk-size=2mb --progress
2
3blob-download[Qxz3EClVN]
4blob-download[xvC3nClSF]
5s3://abc/file-2gb  513 MiB / 2 GiB      [==============>-----------------------------------------------] 24 %
6s3://abc/file-100mb 44.17 MiB / 100 MiB [==========================>-----------------------------------] 44 %