Start, Stop, and monitor distributed parallel sorting (dSort)

1
$ ais dsort --help
2
NAME:
3
   ais dsort - (alias for "job start dsort") start dsort job
4
   Required parameters:
5
              - input_bck: source bucket (used as both source and destination if the latter not specified)
6
              - input_format: (see docs and examples below)
7
              - output_format: (ditto)
8
              - output_shard_size: (as the name implies)
9
   E.g. inline JSON spec:
10
                $ ais start dsort '{
11
                  "extension": ".tar",
12
                  "input_bck": {"name": "dsort-testing"},
13
                  "input_format": {"template": "shard-{0..9}"},
14
                  "output_shard_size": "200KB",
15
                  "description": "pack records into categorized shards",
16
                  "ekm_file": "http://website.web/static/ekm_file.txt",
17
                  "ekm_file_sep": " "
18
                }'
19
   E.g. inline YAML spec:
20
                $ ais start dsort -f - <<EOM
21
                  extension: .tar
22
                  input_bck:
23
                      name: dsort-testing
24
                  input_format:
25
                      template: shard-{0..9}
26
                  output_format: new-shard-{0000..1000}
27
                  output_shard_size: 10KB
28
                  description: shuffle shards from 0 to 9
29
                  algorithm:
30
                      kind: shuffle
31
                  EOM
32
   Tip: use '--dry-run' to see the results without making any changes
33
   Tip: use '--verbose' to print the spec (with all its parameters including applied defaults)
34
   See also: docs/dsort.md, docs/cli/dsort.md, and ais/test/scripts/dsort*
35
36
USAGE:
37
   ais dsort [SRC_BUCKET] [DST_BUCKET] --spec [JSON_SPECIFICATION|YAML_SPECIFICATION|-] [command options]
38
39
OPTIONS:
40
   --spec value, -f value  path to JSON or YAML request specification
41
   --verbose, -v           verbose
42
   --help, -h              show help

1
$ ais start dsort ais://src ais://dst -f ais/test/scripts/dsort-ex1-spec.json --verbose
2
PROPERTY                         VALUE
3
algorithm.content_key_type       -
4
algorithm.decreasing             false
5
algorithm.extension              -
6
algorithm.kind                   alphanumeric
7
algorithm.seed                   -
8
create_concurrency_max_limit     0
9
description                      sort shards alphanumerically
10
dry_run                          false
11
dsorter_type                     -
12
extension                        .tar
13
extract_concurrency_max_limit    0
14
input_bck                        ais://src
15
input_format.objnames            -
16
input_format.template            shard-{0..9}
17
max_mem_usage                    -
18
ekm_file                       -
19
ekm_file_sep                   \t
20
output_bck                       ais://dst
21
output_format                    new-shard-{0000..1000}
22
output_shard_size                10KB
23
24
Config override:                 none
25
26
srt-M8ld-VU_i

1
$ ais start dsort -f dsort_spec.json
2
JGHEoo89gg

1
$ ais start dsort -f - <<EOM
2
extension: .tar
3
input_bck:
4
    name: dsort-testing
5
input_format:
6
    template: shard-{0..9}
7
output_format: new-shard-{0000..1000}
8
output_shard_size: 10KB
9
description: shuffle shards from 0 to 9
10
algorithm:
11
    kind: shuffle
12
EOM
13
JGHEoo89gg

1
$ ais start dsort --spec '{
2
    "extension": ".tar",
3
    "input_bck": {"name": "dsort-testing"},
4
    "input_format": {"template": "shard-{0..9}"},
5
    "output_shard_size": "200KB",
6
    "description": "pack records into categorized shards",
7
    "ekm_file": "http://website.web/static/ekm_file.txt",
8
    "ekm_file_sep": " "
9
}'
10
JGHEoo89gg

1
$ ais show job dsort --help
2
3
NAME:
4
   ais show job - Show running and/or finished jobs,
5
     e.g.:
6
     - show job tco-cysbohAGL              - show a given (multi-object copy/transform) job identified by its unique ID;
7
     - show job copy-listrange             - show all running multi-object copies;
8
     - show job copy-objects               - same as above (using display name);
9
     - show job copy                       - show all copying jobs including both bucket-to-bucket and multi-object;
10
     - show job copy-objects --all         - show both running and already finished (or stopped) multi-object copies;
11
     - show job list                       - show all running list-objects jobs;
12
     - show job ls                         - same as above;
13
     - show job ls --refresh 10            - same as above with periodic _refreshing_ every 10 seconds;
14
     - show job ls --refresh 10 --count 4  - same as above but only for the first four 10-seconds intervals;
15
     - show job prefetch-listrange         - show all running prefetch jobs;
16
     - show job prefetch                   - same as above;
17
     - show job prefetch --refresh 1m      - show all running prefetch jobs at 1 minute intervals (until Ctrl-C);
18
     - show job evict                      - all running bucket and/or data evicting jobs;
19
     - show job --all                      - show absolutely all jobs, running and finished.
20
21
USAGE:
22
   ais show job [NAME] [JOB_ID] [NODE_ID] [BUCKET] [command options]
23
24
OPTIONS:
25
   --all             Include all jobs: running, finished, and aborted
26
   --count value     Used together with '--refresh' to limit the number of generated reports, e.g.:
27
                      '--refresh 10 --count 5' - run 5 times with 10s interval (default: 0)
28
   --date-time       Override the default hh:mm:ss (hours, minutes, seconds) time format - include calendar date as well
29
   --json, -j        JSON input/output
30
   --log value       Filename to log metrics (statistics)
31
   --no-headers, -H  Display tables without headers
32
   --progress        Show progress bar(s) and progress of execution in real time
33
   --refresh value   Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
34
                     valid time units: ns, us (or µs), ms, s (default), m, h
35
   --regex value     Regular expression to select jobs by name, kind, or description, e.g.: --regex "ec|mirror|elect"
36
   --units value     Show statistics and/or parse command-line specified sizes using one of the following units of measurement:
37
                     iec - IEC format, e.g.: KiB, MiB, GiB (default)
38
                     si  - SI (metric) format, e.g.: KB, MB, GB
39
                     raw - do not convert to (or from) human-readable format
40
   --verbose, -v     Show extended statistics
41
   --help, -h        Show help

1
$ ais show job dsort --regex "^sort (.*)"
2
JOB ID		 STATUS		 START		 FINISH			 DESCRIPTION
3
nro_Y5h9n	 Finished	 03-16 11:39:07	 03-16 11:39:07 	 sort shards from 0 to 9
4
Key_Y5h9n	 Finished	 03-16 11:39:23	 03-16 11:39:23 	 sort shards from 10 to 19
5
enq9Y5Aqn	 Finished	 03-16 11:39:34	 03-16 11:39:34 	 sort shards from 20 to 29

1
$ ais show job dsort 5JjIuGemR --refresh 500ms --log "/tmp/dsort_run.txt"
2
Dsort job has finished successfully in 21.948806ms:
3
  Longest extraction:	1.49907ms
4
  Longest sorting:	8.288299ms
5
  Longest creation:	4.553µs

1
$ ais show job dsort 5JjIuGemR --json
2
{
3
  "825090t8089": {
4
    "local_extraction": {
5
      "started_time": "2020-05-28T09:53:42.466267891-04:00",
6
      "end_time": "2020-05-28T09:53:42.50773835-04:00",
7
      ....
8
     },
9
     ....
10
  },
11
  ....
12
}

1
$ ais show job dsort 5JjIuGemR 766516t8087 --json
2
{
3
  "766516t8087": {
4
    "local_extraction": {
5
      "started_time": "2020-05-28T09:53:42.466267891-04:00",
6
      "end_time": "2020-05-28T09:53:42.50773835-04:00",
7
      ....
8
     },
9
     ....
10
  }
11
}

1
$ ais show job dsort 5JjIuGemR --json | jq .[].meta_sorting.running
2
false
3
false
4
true
5
false

1
$ ais show job dsort 5JjIuGemR --json | jq 'to_entries[] | [.key, .value.shard_creation.created_count]'
2
[
3
  "766516t8087",
4
  "189"
5
]
6
[
7
  "710650t8086",
8
  "207"
9
]
10
[
11
  "825090t8089",
12
  "211"
13
]
14
[
15
  "743838t8088",
16
  "186"
17
]
18
[
19
  "354275t8085",
20
  "207"
21
]

1
$ ais wait --help
2
3
NAME:
4
   ais wait - (alias for "job wait") wait for a specific batch job to complete (press <TAB-TAB> to select, '--help' for more options)
5
6
USAGE:
7
   ais wait [NAME] [JOB_ID] [NODE_ID] [BUCKET] [command options]
8
9
OPTIONS:
10
   --progress       Show progress bar(s) and progress of execution in real time
11
   --refresh value  Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
12
                    valid time units: ns, us (or µs), ms, s (default), m, h
13
   --timeout value  Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
14
                    valid time units: ns, us (or µs), ms, s (default), m, h
15
   --help, -h       Show help