Start, Stop, and monitor downloads

1
$ ais download --help
2
NAME:
3
   ais download - (alias for "job start download") Download files and objects from remote sources, e.g.:
4
     - 'ais download http://example.com/file.tar ais://bucket/'                            - download from HTTP into AIS bucket;
5
     - 'ais download s3://bucket/file.tar ais://local-bucket/'                             - download from S3 into AIS bucket;
6
     - 'ais download gs://bucket/file.tar ais://local/ --progress'                         - download from GCS with progress monitoring;
7
     - 'ais download s3://bucket ais://local --sync'                                       - download and sync entire S3 bucket;
8
     - 'ais download "gs://bucket/file-{001..100}.tar" ais://local/'                       - download range of files using template;
9
     - 'ais download --hf-model bert-base-uncased --hf-file config.json ais://local/'      - download specific file from HuggingFace model;
10
     - 'ais download --hf-dataset squad --hf-file train-v1.1.json ais://local/ --hf-auth'  - download dataset file with authentication;
11
     - 'ais download --hf-model bert-large-uncased ais://local/ --blob-threshold 100MB'    - download model with size-based routing (large files get individual jobs).
12
13
USAGE:
14
   ais download SOURCE DESTINATION [command options]
15
16
OPTIONS:
17
   blob-threshold     Utilize built-in blob-downloader for remote objects greater than the specified (threshold) size
18
                      in IEC or SI units, or "raw" bytes (e.g.: 4mb, 1MiB, 1048576, 128k; see '--units')
19
   description,desc   job description
20
   download-timeout   Server-side time limit for downloading a single file from remote source;
21
                      valid time units: ns, us (or µs), ms, s (default), m, h
22
   hf-auth            HuggingFace authentication token for private repositories
23
   hf-dataset         HuggingFace dataset repository name, e.g.:
24
                      --hf-dataset squad
25
                      --hf-dataset glue
26
                      --hf-dataset lhoestq/demo1
27
   hf-file            Specific file to download from HF repository (optional, downloads entire repository if not specified)
28
   hf-model           HuggingFace model repository name, e.g.:
29
                      --hf-model bert-base-uncased
30
                      --hf-model microsoft/DialoGPT-medium
31
                      --hf-model openai/whisper-large-v2
32
   hf-revision        HuggingFace repository revision/branch/tag (default: main)
33
   limit-bph          Maximum download speed, or more exactly: maximum download size per target (node) per hour, e.g.:
34
                      '--limit-bph 1GiB' (or same: '--limit-bph 1073741824');
35
                      the value is parsed in accordance with the '--units' (see '--units' for details);
36
                      omitting the flag or specifying '--limit-bph 0' means that download won't be throttled
37
   max-conns          Maximum number of connections each target can make concurrently (up to num mountpaths)
38
   object-list,from   Path to file containing JSON array of object names to download
39
   progress           Show progress bar(s) and progress of execution in real time
40
   progress-interval  Download progress interval for continuous monitoring;
41
                      valid time units: ns, us (or µs), ms, s (default), m, h
42
   sync               Fully synchronize in-cluster content of a given remote bucket with its (Cloud or remote AIS) source;
43
                      the option is, effectively, a stronger variant of the '--latest' (option):
44
                      in addition to bringing existing in-cluster objects in-sync with their respective out-of-band updates (if any)
45
                      it also entails removing in-cluster objects that are no longer present remotely;
46
                      like '--latest', this option provides operation-level control over synchronization
47
                      without requiring to change the corresponding bucket configuration: 'versioning.synchronize';
48
                      see also:
49
                        - 'ais show bucket BUCKET versioning'
50
                        - 'ais bucket props set BUCKET versioning'
51
                        - 'ais start rebalance'
52
                        - 'ais ls --check-versions'
53
   timeout            Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
54
                      valid time units: ns, us (or µs), ms, s (default), m, h
55
   units              Show statistics and/or parse command-line specified sizes using one of the following units of measurement:
56
                      iec - IEC format, e.g.: KiB, MiB, GiB (default)
57
                      si  - SI (metric) format, e.g.: KB, MB, GB
58
                      raw - do not convert to (or from) human-readable format
59
   wait               Wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
60
   help, h            Show help

1
$ ais show job download QdwOYMAqg
2
Download progress: 64/141 (45.39%)
3
Errors (10) occurred during the download. To see detailed info run `ais show job download QdwOYMAqg -v`
4
$ ais show job download QdwOYMAqg -v
5
Download progress: 64/141 (45.39%)
6
Progress of files that are currently being downloaded:
7
	imagenet/imagenet_train-000002.tgz: 16.39MiB/946.91MiB (1.73%)
8
	imagenet/imagenet_train-000023.tgz: 113.81MiB/946.35MiB (12.03%)
9
	...
10
Errors:
11
	imagenet/imagenet_train-000049.tgz: request failed with 404 status code (Not Found)
12
	imagenet/imagenet_train-000123.tgz: request failed with 404 status code (Not Found)
13
	...

1
$ ais show job download QdwOYMAqg
2
Done: 120 files downloaded, 21 errors
3
$ ais show job download QdwOYMAqg -v
4
Done: 120 files downloaded, 21 errors
5
Errors:
6
	imagenet/imagenet_train-000049.tgz: request failed with 404 status code (Not Found)
7
	imagenet/imagenet_train-000123.tgz: request failed with 404 status code (Not Found)
8
	...

1
$ ais bucket props set ais://lpr-vision-copy backend_bck=gcp://lpr-vision
2
Bucket props successfully updated
3
"backend_bck.name" set to:"lpr-vision" (was:"")
4
"backend_bck.provider" set to:"gcp" (was:"")
5
$ ais start download gs://lpr-vision ais://lpr-vision-copy
6
QdwOYMAqg
7
Run `ais show job download QdwOYMAqg` to monitor the progress of downloading.

1
$ ais ls --no-headers gcp://lpr-vision | wc -l
2
50
3
$ ais bucket props set ais://lpr-vision-copy backend_bck=gcp://lpr-vision
4
Bucket props successfully updated
5
"backend_bck.name" set to:"lpr-vision" (was:"")
6
"backend_bck.provider" set to:"gcp" (was:"")
7
$ ais ls --no-headers --cached ais://lpr-vision-copy | wc -l
8
0
9
$ ais start download gs://lpr-vision ais://lpr-vision-copy
10
QdwOYMAqg
11
Run `ais show job download QdwOYMAqg` to monitor the progress of downloading.
12
$ ais wait download QdwOYMAqg
13
$ ais ls --no-headers --cached ais://lpr-vision-copy | wc -l
14
50
15
$ # Remove some objects from `gcp://lpr-vision`
16
$ ais ls --no-headers gcp://lpr-vision | wc -l
17
40
18
$ ais ls --no-headers --cached ais://lpr-vision-copy | wc -l
19
50
20
$ ais start download --sync gs://lpr-vision ais://lpr-vision-copy
21
fjwiIEMfa
22
Run `ais show job download fjwiIEMfa` to monitor the progress of downloading.
23
$ ais wait download fjwiIEMfa
24
$ ais ls --no-headers --cached ais://lpr-vision-copy | wc -l
25
40
26
$ diff <(ais ls gcp://lpr-vision) <(ais ls --cached ais://lpr-vision-copy) | wc -l
27
0

1
$ ais bucket props set ais://lpr-vision-copy backend_bck=gcp://lpr-vision
2
Bucket props successfully updated
3
"backend_bck.name" set to:"lpr-vision" (was:"")
4
"backend_bck.provider" set to:"gcp" (was:"")
5
$ ais start download gs://lpr-vision/dir/prefix- ais://lpr-vision-copy
6
QdwOYMAqg
7
Run `ais show job download QdwOYMAqg` to monitor the progress of downloading.

1
$ ais show job download 5JjIuGemR --progress --refresh 500ms
2
Files downloaded:                              0/141 [--------------------------------------------------------------] 0 %
3
imagenet/imagenet_train-000006.tgz 192.7MiB/947.0MiB [============>-------------------------------------------------| 00:08:52 ]   1.4 MiB/s
4
imagenet/imagenet_train-000015.tgz 238.8MiB/946.3MiB [===============>----------------------------------------------| 00:05:42 ]   2.1 MiB/s
5
imagenet/imagenet_train-000022.tgz  31.2MiB/946.5MiB [=>------------------------------------------------------------| 00:24:35 ] 703.1 KiB/s
6
imagenet/imagenet_train-000043.tgz  38.5MiB/945.9MiB [==>-----------------------------------------------------------| 00:12:50 ]   1.1 MiB/s

1
$ ais show job download --regex "^downloads (.*)"
2
JOB ID		 STATUS		 ERRORS	 DESCRIPTION
3
cudIYMAqg	 Finished	 0	 downloads whole imagenet bucket
4
fjwiIEMfa	 Finished	 0	 downloads range lpr-bucket from gcp://lpr-bucket