Storage and mountpath management

ais storage command supports the following subcommands:

1 $ ais storage <TAB-TAB>
2 cleanup     disk        mountpath   summary     validate

Alternatively (or in addition), run with --help to view subcommands and short descriptions, both:

1 $ ais storage --help
2 NAME:
3    ais storage - monitor and manage clustered storage
4 
5 USAGE:
6    ais storage command [arguments...] [command options]
7 
8 COMMANDS:
9    show       show storage usage and utilization, disks and mountpaths
10    summary    show bucket sizes and %% of used capacity on a per-bucket basis
11    validate   check buckets for misplaced objects and objects that have insufficient numbers of copies or EC slices
12    mountpath  show and attach/detach target mountpaths
13    disk       show disk utilization and read/write statistics
14    cleanup    perform storage cleanup: remove deleted objects and old/obsolete workfiles
15 
16 OPTIONS:
17    --help, -h  show help

As always, each subcommand (above) will have its own help and usage examples - the latter possibly spread across multiple markdowns.

You can easily look up examples and descriptions of any keyword via a simple find, for instance:

1 $ find . -type f -name "*.md" | xargs grep "ais.*mountpath"

Storage cleanup

1 $ ais storage cleanup --help
2 NAME:
3    ais storage cleanup - remove deleted objects and old/obsolete workfiles; remove misplaced objects; optionally, remove zero size objects
4 
5 USAGE:
6    ais storage cleanup PROVIDER:[//BUCKET_NAME] [command options]
7 
8 OPTIONS:
9    --force, -f      disregard interrupted rebalance and possibly other conditions preventing full cleanup
10                     (tip: check 'ais config cluster lru.dont_evict_time' as well)
11    --rm-zero-size   remove zero-size objects (caution: advanced usage only)
12    --wait           wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
13    --timeout value  maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
14                     valid time units: ns, us (or µs), ms, s (default), m, h
15    --help, -h       show help

Similar to all supported batch operations (aka xactions), cleanup runs asynchronously and can be monitored during its run, e.g.:

1 # ais storage cleanup
2 Started storage cleanup "BlpmlObF8", use 'ais job show xaction BlpmlObF8' to monitor the progress

Further references:

Show capacity usage

For command line options and usage examples, please refer to:

bucket summary

Validate in-cluster content for misplaced objects and missing copies

1 $ ais scrub --help
2 
3 NAME:
4    ais scrub - (alias for "storage validate") Check in-cluster content for misplaced objects, objects that have insufficient numbers of copies, zero size, and more
5    e.g.:
6      * ais storage validate                 - validate all in-cluster buckets;
7      * ais scrub                            - same as above;
8      * ais storage validate ais             - validate (a.k.a. scrub) all ais:// buckets;
9      * ais scrub s3                         - ditto, all s3:// buckets;
10      * ais scrub s3 --refresh 10            - same as above while refreshing runtime counter(s) every 10s;
11      * ais scrub gs://abc/images/           - validate part of the gcp bucket under 'images/`;
12      * ais scrub gs://abc --prefix images/  - same as above.
13 
14 USAGE:
15    ais scrub [BUCKET[/PREFIX]] [PROVIDER] [command options]
16 
17 OPTIONS:
18    --all-columns          Show all columns, including those with only zero values
19    --cached               Only visit in-cluster objects, i.e., objects from the respective remote bucket that are present ("cached") in the cluster
20    --count value          Used together with '--refresh' to limit the number of generated reports, e.g.:
21                            '--refresh 10 --count 5' - run 5 times with 10s interval (default: 0)
22    --large-size value     Count and report all objects that are larger or equal in size  (e.g.: 4mb, 1MiB, 1048576, 128k; default: 5 GiB)
23    --limit value          The maximum number of objects to list, get, or otherwise handle (0 - unlimited; see also '--max-pages'),
24                           e.g.:
25                           - 'ais ls gs://abc/dir --limit 1234 --cached --props size,custom,atime'  - list no more than 1234 objects
26                           - 'ais get gs://abc /dev/null --prefix dir --limit 1234'                 - get --/--
27                           - 'ais scrub gs://abc/dir --limit 1234'                                  - scrub --/-- (default: 0)
28    --max-pages value      Maximum number of pages to display (see also '--page-size' and '--limit')
29                           e.g.: 'ais ls az://abc --paged --page-size 123 --max-pages 7 (default: 0)
30    --no-headers, -H       Display tables without headers
31    --non-recursive, --nr  Non-recursive operation, e.g.:
32                           - 'ais ls gs://bucket/prefix --nr'   - list objects and/or virtual subdirectories with names starting with the specified prefix;
33                           - 'ais ls gs://bucket/prefix/ --nr'  - list contained objects and/or immediately nested virtual subdirectories _without_ recursing into the latter;
34                           - 'ais prefetch s3://bck/abcd --nr'  - prefetch a single named object (see 'ais prefetch --help' for details);
35                           - 'ais rmo gs://bucket/prefix --nr'  - remove a single object with the specified name (see 'ais rmo --help' for details)
36    --page-size value      Maximum number of object names per page; when the flag is omitted or 0
37                           the maximum is defined by the corresponding backend; see also '--max-pages' and '--paged' (default: 0)
38    --prefix value         For each bucket, select only those objects (names) that start with the specified prefix, e.g.:
39                           '--prefix a/b/c' - sum up sizes of the virtual directory a/b/c and objects from the virtual directory
40                           a/b that have names (relative to this directory) starting with the letter c
41    --refresh value        Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
42                           valid time units: ns, us (or µs), ms, s (default), m, h
43    --small-size value     Count and report all objects that are smaller or equal in size (e.g.: 4, 4b, 1k, 128kib; default: 0)
44    --help, -h             Show help

Checks all objects of the bucket BUCKET and show number of misplaced objects, number of objects that have insufficient number of copies, etc.

If optional arguments are omitted, show information about all in-cluster buckets.

Example: validate a given prefix-defined portion of an s3 bucket

1 $ ais scrub s3://data/my-prefix --large-size 500k
2 
3 BUCKET/PREFIX        OBJECTS         NOT-CACHED      SMALL   LARGE           VER-CHANGED     DELETED
4 s3://data/my-prefix  1637 (1.6GiB)   1465 (1.4GiB)   -       172 (172.0MiB)  1 (1.0MiB)      1 (1.0MiB)
5 
6 Detailed Logs
7 -------------
8 * not-cached objects:   /tmp/.ais-scrub-not-cached.204f71.log (1465 records)
9 * large objects:        /tmp/.ais-scrub-large.204f71.log (172 records)
10 * ver-changed objects:  /tmp/.ais-scrub-ver-changed.204f71.log (1 record)
11 * deleted objects:      /tmp/.ais-scrub-deleted.204f71.log (1 record)

Example: same as above but show all columns

In other words, include relevant metrics that have only zero values.

1 $ ais scrub s3://data/my-prefix --large-size 500k --all-columns
2 
3 BUCKET/PREFIX        OBJECTS         NOT-CACHED      MISPLACED(cluster)  MISPLACED(mountpath) MISSING-COPIES  SMALL  LARGE          VER-CHANGED   DELETED
4 s3://data/my-prefix  1637 (1.6GiB)   1465 (1.4GiB)   -                   -                    -               -      172 (172.0MiB) 1 (1.0MiB)    1 (1.0MiB)
5 
6 Detailed Logs
7 -------------
8 * not-cached objects:   /tmp/.ais-scrub-not-cached.204f8c.log (1465 records)
9 * large objects:        /tmp/.ais-scrub-large.204f8c.log (172 records)
10 * ver-changed objects:  /tmp/.ais-scrub-ver-changed.204f8c.log (1 record)
11 * deleted objects:      /tmp/.ais-scrub-deleted.204f8c.log (1 record)

Note that 172 (records) = 1637 - 1465.

Mountpath (and disk) management

There are two related commands:

ais storage disk
ais storage mountpath

where mountpath is a higher-level abstraction that typically “utilizes” a single undivided disk. More exactly:

A mountpath is a single disk or a volume (a RAID) formatted with a local filesystem of choice, and a local directory that AIS utilizes to store user data and AIS metadata. A mountpath can be disabled and (re)enabled, automatically or administratively, at any point during runtime. In a given cluster, a total number of mountpaths would normally compute as a direct product of (number of storage targets) x (number of disks in each target).

You can manage and monitor (i.e., show) disks and mountpaths using ais storage command.

For strictly monitoring purposes, you can universally use ais show command, e.g.: ais show storage disk, etc.

Show disks

ais storage disk show [TARGET_ID]

or, same:

ais show storage disk [TARGET_ID]

Show mountpaths

As the name implies, the syntax:

ais show storage mountpath [TARGET_ID]

for example:

1 $ ais show storage mountpath t[TqPtghbiRw]
2 
3 TqPtghbiRw
4         Used Capacity (all disks): avg 15% max 18%
5                                                 /ais/mp1/2 /dev/nvme0n1(xfs)
6                                                 /ais/mp2/2 /dev/nvme1n1(xfs)
7                                                 /ais/mp3/2 /dev/nvme2n1(xfs)
8                                                 /ais/mp4/2 /dev/nvme3n1(xfs)

As always, --help will also list supported options. Note in particular the option to run continuously and periodically:

1 $ ais show storage mountpath --help
2 NAME:
3    ais show storage mountpath - show target mountpaths
4 
5 USAGE:
6    ais show storage mountpath [TARGET_ID] [command options]
7 
8 OPTIONS:
9    --refresh value  interval for continuous monitoring;
10                     valid time units: ns, us (or µs), ms, s (default), m, h
11    --count value    used together with '--refresh' to limit the number of generated reports (default: 0)
12    --json, -j       json input/output
13    --help, -h       show help

Show mountpaths for a given target or all targets.

Ease of Usage notice: like all other ais show commands, ais show storage mountpath is an alias (or a shortcut) - in this specific case - for ais storage mountpath show.

Examples (slightly outdated)

1 $ ais storage mountpath show 12356t8085
2 247389t8085
3         Available:
4 			/tmp/ais/5/3
5 			/tmp/ais/5/1
6         Disabled:
7 			/tmp/ais/5/2
8 
9 $ ais storage mountpath show
10 12356t8085
11         Available:
12 			/tmp/ais/5/3
13 			/tmp/ais/5/1
14         Disabled:
15 			/tmp/ais/5/2
16 147665t8084
17         Available:
18 			/tmp/ais/4/3
19 			/tmp/ais/4/1
20 			/tmp/ais/4/2
21 426988t8086
22 	No mountpaths

Attach mountpath

ais storage mountpath attach TARGET_ID=MOUNTPATH [DAEMONID=MOUNTPATH...]

Attach a mountpath on a specified target to AIS storage.

Examples

1 $ ais storage mountpath attach 12367t8080=/data/dir

Detach mountpath

ais storage mountpath detach TARGET_ID=MOUNTPATH [DAEMONID=MOUNTPATH...]

Detach a mountpath on a specified target from AIS storage.

Examples

1 $ ais storage mountpath detach 12367t8080=/data/dir

Storage and mountpath management

Storage and mountpath management

Table of Contents

Storage cleanup

Show capacity usage

Validate in-cluster content for misplaced objects and missing copies

Example: validate a given prefix-defined portion of an s3 bucket

Example: same as above but show all columns

Mountpath (and disk) management

Show disks

Show mountpaths

Examples (slightly outdated)

Attach mountpath

Examples

Detach mountpath

Examples

Table of Contents

Storage cleanup

Show capacity usage

Validate in-cluster content for misplaced objects and missing copies

Example: validate a given prefix-defined portion of an s3 bucket

Example: same as above but show all columns

Mountpath (and disk) management

Show disks

Show mountpaths

Examples (slightly outdated)

Attach mountpath

Examples

Detach mountpath

Examples

1	$ ais storage <TAB-TAB>
2	cleanup disk mountpath summary validate

1	$ ais storage --help
2	NAME:
3	ais storage - monitor and manage clustered storage
4
5	USAGE:
6	ais storage command [arguments...] [command options]
7
8	COMMANDS:
9	show show storage usage and utilization, disks and mountpaths
10	summary show bucket sizes and %% of used capacity on a per-bucket basis
11	validate check buckets for misplaced objects and objects that have insufficient numbers of copies or EC slices
12	mountpath show and attach/detach target mountpaths
13	disk show disk utilization and read/write statistics
14	cleanup perform storage cleanup: remove deleted objects and old/obsolete workfiles
15
16	OPTIONS:
17	--help, -h show help

1	$ ais storage cleanup --help
2	NAME:
3	ais storage cleanup - remove deleted objects and old/obsolete workfiles; remove misplaced objects; optionally, remove zero size objects
4
5	USAGE:
6	ais storage cleanup PROVIDER:[//BUCKET_NAME] [command options]
7
8	OPTIONS:
9	--force, -f disregard interrupted rebalance and possibly other conditions preventing full cleanup
10	(tip: check 'ais config cluster lru.dont_evict_time' as well)
11	--rm-zero-size remove zero-size objects (caution: advanced usage only)
12	--wait wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
13	--timeout value maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
14	valid time units: ns, us (or µs), ms, s (default), m, h
15	--help, -h show help

1	# ais storage cleanup
2	Started storage cleanup "BlpmlObF8", use 'ais job show xaction BlpmlObF8' to monitor the progress

1	$ ais scrub --help
2
3	NAME:
4	ais scrub - (alias for "storage validate") Check in-cluster content for misplaced objects, objects that have insufficient numbers of copies, zero size, and more
5	e.g.:
6	* ais storage validate - validate all in-cluster buckets;
7	* ais scrub - same as above;
8	* ais storage validate ais - validate (a.k.a. scrub) all ais:// buckets;
9	* ais scrub s3 - ditto, all s3:// buckets;
10	* ais scrub s3 --refresh 10 - same as above while refreshing runtime counter(s) every 10s;
11	* ais scrub gs://abc/images/ - validate part of the gcp bucket under 'images/`;
12	* ais scrub gs://abc --prefix images/ - same as above.
13
14	USAGE:
15	ais scrub [BUCKET[/PREFIX]] [PROVIDER] [command options]
16
17	OPTIONS:
18	--all-columns Show all columns, including those with only zero values
19	--cached Only visit in-cluster objects, i.e., objects from the respective remote bucket that are present ("cached") in the cluster
20	--count value Used together with '--refresh' to limit the number of generated reports, e.g.:
21	'--refresh 10 --count 5' - run 5 times with 10s interval (default: 0)
22	--large-size value Count and report all objects that are larger or equal in size (e.g.: 4mb, 1MiB, 1048576, 128k; default: 5 GiB)
23	--limit value The maximum number of objects to list, get, or otherwise handle (0 - unlimited; see also '--max-pages'),
24	e.g.:
25	- 'ais ls gs://abc/dir --limit 1234 --cached --props size,custom,atime' - list no more than 1234 objects
26	- 'ais get gs://abc /dev/null --prefix dir --limit 1234' - get --/--
27	- 'ais scrub gs://abc/dir --limit 1234' - scrub --/-- (default: 0)
28	--max-pages value Maximum number of pages to display (see also '--page-size' and '--limit')
29	e.g.: 'ais ls az://abc --paged --page-size 123 --max-pages 7 (default: 0)
30	--no-headers, -H Display tables without headers
31	--non-recursive, --nr Non-recursive operation, e.g.:
32	- 'ais ls gs://bucket/prefix --nr' - list objects and/or virtual subdirectories with names starting with the specified prefix;
33	- 'ais ls gs://bucket/prefix/ --nr' - list contained objects and/or immediately nested virtual subdirectories _without_ recursing into the latter;
34	- 'ais prefetch s3://bck/abcd --nr' - prefetch a single named object (see 'ais prefetch --help' for details);
35	- 'ais rmo gs://bucket/prefix --nr' - remove a single object with the specified name (see 'ais rmo --help' for details)
36	--page-size value Maximum number of object names per page; when the flag is omitted or 0
37	the maximum is defined by the corresponding backend; see also '--max-pages' and '--paged' (default: 0)
38	--prefix value For each bucket, select only those objects (names) that start with the specified prefix, e.g.:
39	'--prefix a/b/c' - sum up sizes of the virtual directory a/b/c and objects from the virtual directory
40	a/b that have names (relative to this directory) starting with the letter c
41	--refresh value Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
42	valid time units: ns, us (or µs), ms, s (default), m, h
43	--small-size value Count and report all objects that are smaller or equal in size (e.g.: 4, 4b, 1k, 128kib; default: 0)
44	--help, -h Show help

1	$ ais scrub s3://data/my-prefix --large-size 500k
2
3	BUCKET/PREFIX OBJECTS NOT-CACHED SMALL LARGE VER-CHANGED DELETED
4	s3://data/my-prefix 1637 (1.6GiB) 1465 (1.4GiB) - 172 (172.0MiB) 1 (1.0MiB) 1 (1.0MiB)
5
6	Detailed Logs
7	-------------
8	* not-cached objects: /tmp/.ais-scrub-not-cached.204f71.log (1465 records)
9	* large objects: /tmp/.ais-scrub-large.204f71.log (172 records)
10	* ver-changed objects: /tmp/.ais-scrub-ver-changed.204f71.log (1 record)
11	* deleted objects: /tmp/.ais-scrub-deleted.204f71.log (1 record)

1	$ ais scrub s3://data/my-prefix --large-size 500k --all-columns
2
3	BUCKET/PREFIX OBJECTS NOT-CACHED MISPLACED(cluster) MISPLACED(mountpath) MISSING-COPIES SMALL LARGE VER-CHANGED DELETED
4	s3://data/my-prefix 1637 (1.6GiB) 1465 (1.4GiB) - - - - 172 (172.0MiB) 1 (1.0MiB) 1 (1.0MiB)
5
6	Detailed Logs
7	-------------
8	* not-cached objects: /tmp/.ais-scrub-not-cached.204f8c.log (1465 records)
9	* large objects: /tmp/.ais-scrub-large.204f8c.log (172 records)
10	* ver-changed objects: /tmp/.ais-scrub-ver-changed.204f8c.log (1 record)
11	* deleted objects: /tmp/.ais-scrub-deleted.204f8c.log (1 record)

1	$ ais show storage mountpath t[TqPtghbiRw]
2
3	TqPtghbiRw
4	Used Capacity (all disks): avg 15% max 18%
5	/ais/mp1/2 /dev/nvme0n1(xfs)
6	/ais/mp2/2 /dev/nvme1n1(xfs)
7	/ais/mp3/2 /dev/nvme2n1(xfs)
8	/ais/mp4/2 /dev/nvme3n1(xfs)

1	$ ais show storage mountpath --help
2	NAME:
3	ais show storage mountpath - show target mountpaths
4
5	USAGE:
6	ais show storage mountpath [TARGET_ID] [command options]
7
8	OPTIONS:
9	--refresh value interval for continuous monitoring;
10	valid time units: ns, us (or µs), ms, s (default), m, h
11	--count value used together with '--refresh' to limit the number of generated reports (default: 0)
12	--json, -j json input/output
13	--help, -h show help

1	$ ais storage mountpath show 12356t8085
2	247389t8085
3	Available:
4	/tmp/ais/5/3
5	/tmp/ais/5/1
6	Disabled:
7	/tmp/ais/5/2
8
9	$ ais storage mountpath show
10	12356t8085
11	Available:
12	/tmp/ais/5/3
13	/tmp/ais/5/1
14	Disabled:
15	/tmp/ais/5/2
16	147665t8084
17	Available:
18	/tmp/ais/4/3
19	/tmp/ais/4/1
20	/tmp/ais/4/2
21	426988t8086
22	No mountpaths