Machine‑Learning Operations (ais ml)

1
$ ais ml <TAB><TAB>
2
get-batch         # Consolidate objects / archived files into one archive
3
lhotse-get-batch  # Drive multiple get‑batch runs from Lhotse manifests

1
$ ais ml get-batch --help
2
$ ais ml lhotse-get-batch --help

1
ais ml get-batch [SRC ...] DST_ARCHIVE [flags]
2
3
# SRC can be:
4
#   - bucket or virtual directory      (ais://speech/)
5
#   - single object                    (s3://models/chkpt‑001.pth)
6
#   - archived path inside an object   (ais://data/shard.tar/file.wav)

1
ais ml lhotse-get-batch --cuts manifest.jsonl[.gz | .lz4] [DST] [flags]
2
3
# With --output-template you may omit DST; the template expands per batch.

1
$ ais ml lhotse-get-batch --help
2
NAME:
3
   ais ml lhotse-get-batch - Get multiple objects from Lhotse manifests and package into consolidated archive(s).
4
     Returns TAR by default; supported formats include: .tar, .tgz or .tar.gz, .zip, .tar.lz4.
5
     Supports chunking, filtering, and multi-output generation from Lhotse cut manifests.
6
     Lhotse manifest format: each line contains a single cut JSON object with recording sources;
7
     Lhotse manifest may be plain (`.jsonl`), gzip‑compressed (`.jsonl.gz` / `.gzip`), or LZ4‑compressed (`.jsonl.lz4`).
8
     Examples:
9
     - 'ais ml lhotse-get-batch --cuts manifest.jsonl.gz output.tar'                                        - entire manifest as single TAR;
10
     - 'ais ml lhotse-get-batch --cuts cuts.jsonl --sample-rate 16000 output.tar'                           - with sample rate conversion;
11
     - 'ais ml lhotse-get-batch --cuts m.jsonl.lz4 --batch-size 1000 --output-template "a-{001..999}.tar"'  - generate 999 'a-*.tar' batches (1000 cuts each)
12
     See [Lhotse docs](https://lhotse.readthedocs.io/) for manifest format details.
13
14
USAGE:
15
   ais ml lhotse-get-batch [BUCKET[/NAME_or_TEMPLATE] ...] [DST_ARCHIVE] --spec [JSON_SPECIFICATION|YAML_SPECIFICATION] [command options]
16
17
OPTIONS:
18
   batch-size       number of cuts per output file
19
   cont-on-err      Keep running archiving xaction (job) in presence of errors in any given multi-object transaction
20
   cuts             path to Lhotse cuts.jsonl or cuts.jsonl.gz or cuts.jsonl.lz4
21
   list             Comma-separated list of object or file names, e.g.:
22
                    --list 'o1,o2,o3'
23
                    --list "abc/1.tar, abc/1.cls, abc/1.jpeg"
24
                    or, when listing files and/or directories:
25
                    --list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
26
   non-verbose,nv   Non-verbose (quiet) output, minimized reporting, fewer warnings
27
   omit-src-bck     When set, strip source bucket names from paths inside the archive (ie., use object names only)
28
   output-template  template for multiple output files (e.g. 'batch-{001..999}.tar')
29
   prefix           Select virtual directories or objects with names starting with the specified prefix, e.g.:
30
                    '--prefix a/b/c'   - matches names 'a/b/c/d', 'a/b/cdef', and similar;
31
                    '--prefix a/b/c/'  - only matches objects from the virtual directory a/b/c/
32
   sample-rate      audio sample-rate (Hz); used to convert sample offsets (in seconds) to byte offsets
33
   spec,f           Path to JSON or YAML request specification
34
   streaming        stream the resulting archive prior to finalizing it in memory
35
   template         Template to match object or file names; may contain prefix (that could be empty) with zero or more ranges
36
                    (with optional steps and gaps), e.g.:
37
                    --template "" # (an empty or '*' template matches everything)
38
                    --template 'dir/subdir/'
39
                    --template 'shard-{1000..9999}.tar'
40
                    --template "prefix-{0010..0013..2}-gap-{1..2}-suffix"
41
                    and similarly, when specifying files and directories:
42
                    --template '/home/dir/subdir/'
43
                    --template "/abc/prefix-{0010..9999..2}-suffix"
44
   yes,y            Assume 'yes' to all questions
45
   help, h          Show help