Batch operations that run asynchronously and may take seconds (minutes, hours, etc.) to execute - are called eXtended actions (xactions).
Internally, xaction is an abstraction at the root of the inheritance hierarchy that also contains specific user-visible jobs: copy-bucket, evict-objects, and more.
For the most recently updated list of all supported jobs and their respective compile-time properties, see the source.
All jobs run asynchronously, have start and stop times, and common generic statistics
Further, each and every job kind has its own display name, access permissions, scope (bucket and/or global), and a number of boolean properties - examples including:
Many kinds of jobs can be manually started via generic job API (which’s in turn utilized by the ais start command - see next).
Notable exceptions include electing new primary and listing objects in a given bucket - in both of those cases, there’s a separate, more convenient and intuitive API that does the job, so to speak.
Job starting, stopping (i.e., aborting), and monitoring commands all have equivalent shorter versions. For instance
ais start downloadcan be expressed asais start download, whileais wait copy-bucket Z8WkHxwIrris the same asais wait Z8WkHxwIrr.
Rest of this document covers starting, stopping, and otherwise managing job kinds and specific job instances. For job monitoring, please use ais show job command and its numerous subcommands and options.
xact package README.batch jobsdsort (distributed shuffle)download from any remote sourcerebalanceais job commandHas the following static completions aka subcommands:
and further:
Notice, though, that start, stop, and wait` (verbs) have shorter versions, e.g.:
ais start is a built-in alias for ais job start, and so on.For all configured pre-built and user-defined aliases (aka “shortcuts”), run
ais aliasorais alias --help
ais start <JOB_NAME> [arguments...]
Start a certain job. Some jobs require additional arguments such as bucket name to execute.
Note: job start download|dsort have slightly different options. Please see their documentation for more:
Starts LRU xaction on all nodes
An administrator may choose to run LRU on a subset of buckets. This can be achieved by using the --buckets flag to provide a comma-separated list of buckets, for instance --buckets bck1,gcp://bck2, on which LRU needs to be performed.
Additionally, the --force(-f) option can be used to override the bucket’s lru.enabled property.
Note: To ensure safety, the force flag (-f) only works when a list of buckets is provided.
Re-chunking converts objects between monolithic and chunked representations based on the specified chunking parameters. The job processes objects in the bucket according to the configured threshold:
objsize_limit are stored as monolithic (single file)objsize_limit are split into chunks of chunk_sizeobjsize_limit is 0, chunking is disabled and all objects are restored as monolithicUsage:
Flags:
--chunk-size SIZE - Size of each chunk (e.g., 16MiB, 20mb). Optional: if omitted, uses the bucket’s current chunk_size--objsize-limit SIZE - Object size threshold for chunking (e.g., 50MiB, 100mb); objects >= this size will be chunked. Optional: if omitted, uses the bucket’s current objsize_limit--prefix PREFIX - Only rechunk objects with the specified prefix (can also be embedded in the bucket URI)--wait - Wait for the job to complete before returning--wait-timeout DURATION - Maximum time to wait (e.g., 5m, 1h)--yes, -y - Assume ‘yes’ to all prompts (skip confirmation)Note: If either size argument is missing, you will be prompted to confirm using the bucket’s current configuration.
Examples:
Rechunk using the bucket’s existing chunk configuration (prompts for confirmation):
Rechunk with one explicit flag and one from bucket (prompts for confirmation):
Rechunk all objects with both flags explicitly provided (no prompt):
Rechunk only objects with a specific prefix using embedded prefix in the URI:
Disable chunking and restore all objects as monolithic:
Wait for the rechunk job to complete:
Note: Regardless of
objsize_limitvalue (even when disabled), objects exceeding the bucket’smaxMonolithicSizeconfiguration will be automatically chunked at that size limit for performance and storage management reasons.
See also: bucket properties,
ais bucket props
Stop a single job or multiple jobs.
ais stop download JOB_IDais stop JOB_IDais stop dsort JOB_IDais stop download --all # stop all downloadsais stop copy-bucket ais://abc --all # stop all copy-bucket jobs where the destination bucket is ais://abcais stop resilver t[rt2erGhbr] # ask target t[rt2erGhbr] to stop resilveringand more.
Note: job stop download|dsort have slightly different options. Please see their documentation for more:
Stops currently running LRU eviction.
ais show job [NAME] [JOB_ID] [NODE_ID] [BUCKET] [command options]
You can show jobs by any combination of the optional (filtering) arguments: NAME, JOB_ID, etc..
Use --all option to include finished (or aborted) jobs.
As usual, press <TAB-TAB> to select and see —help` for details.
job show download|dsorthave slightly different options. Please see their documentation for more:
All jobs show the number of processed objects(column OBJECTS) and the total size of the data(column BYTES).
Both values are cumulative for the entire job’s life-time.
Certain kinds of supported jobs provide extended statistics, including:
The output contains a few extra columns:
ERRORS - the total number of objects EC failed to encodeQUEUE - the average length of working queue: the average number of objects waiting in the queue when a new EC encode request received. Values close to 0 mean that every object was processed immediately after the request had been receivedAVG TIME - the average total processing time for an object: from the moment the object is put to the working queue and to the moment the last encoded slice is sent to another targetENC TIME - the average amount of time spent on encoding an object.The extended statistics may give a hint what is the possible bottleneck:
QUEUE - EC is congested and does not have time to process all incoming requestsQUEUE and ENC TIME, but high ones in AVG TIME may mean that the network is slow and a lot of time spent on sending the encoded slicesQUEUE, and ENC TIME close to AVG TIME may mean that the local hardware is overloaded: either local drives or CPUs are overloaded.Show information about EC restore requests.
The output contains a few extra columns:
ERRORS - the total number of objects EC failed to restoreQUEUE - the average length of working queue: the average number of objects waiting in the queue when a new EC encode request received. Values close to 0 mean that every object was processed immediately after the request had been receivedAVG TIME - the average total processing time for an object: from the moment the object is put to the working queue and to the moment the last encoded slice is sent to another target--top flagShow 2 most recent list jobs:
Show 1 most recent job of any type:
Show 2 most recent list jobs including finished ones:
Certain extended actions have additional CLI. In particular, rebalance stats can also be displayed using the following command:
ais show rebalance
Display details about the most recent rebalance xaction.
Output of this command differs from the generic xaction output.
Default compact tabular view:
Verbose tabular view:
ais wait [NAME] [JOB_ID] [NODE_ID] [BUCKET]
Wait for the specified job to finish.
job wait download|dsorthave slightly different options. Please see their documentation for more:
ais start dsort
Run dSort. Further reference for this command can be found here.
ais start download or ais start download
Run the AIS Downloader. Further reference for this command can be found here.