16.1. Pipeline Load Generator CLI (cpload)

cpload is a CLI tool which allows to specify the pipeline(s) used to create the jobs, the data set(s) used as input for the jobs, and other options for controlling the number of jobs to create, the frequency at which to create them etc.

16.1.1. Requirement:

clara CLI must be installed on the system to use cpload CLI.

16.1.2. Syntax: generate subcommand

cpload generate [(-r|--recurse)] [(-t|--target) <target>] [(-p|--priority) <priority>] [(-n|--count) <number-of-jobs> [(-s|--sequence)] [-w|--delay <delay-seconds>]] <name> <pipeline-id> [-- <data-path>]

Creates a new job using <pipeline-id> with a priority of <priority> and the name of <name>. Pushes the contents of <data-path> to <target> as input for the new job. When multiple jobs are created the name of the job will be expanded to "cpload-<name>-<count>" where count is the ordinal of the job created, starting at one.

<pipeline-id> is expected to be the valid identity of a pipeline known by <target>.

When -- <data-path> is not provided, the job is assumed to have no payload requirement; and no data is uploaded to the payload prior to starting the job.

When <data-path> is a directory, each child file is used as a separate job input. If <data-path> contains child sub-folders along with child files and (-r|--recurse) is provided each immediate child is used as a separate job input; if (-r|--recurse) is not provided an error occurs and no work is generated.

When <data-path> is a file and (-r|--recurse) is provided, an error occurs.

<target> is assumed to be localhost, or the current remote set by cpload logon, unless (-t|--target) <target> is provided.

<priority> is assumed to be normal, unless (-p|--priority) <priority> is provided. Valid priority values are low, normal, high, or immediate.

<number-of-jobs> will be created on <target>. <number-of-jobs> is assumed to be 1 unless (-n|--count) <number-of-jobs> is provided.

When <number-of-jobs> is greater than 1, <data-path> is a directory, and (-s|--sequence) is provided, <data-path> must only contain a series of folders which are named in sequence (i.e. data/1, data/2, data/3, etc.). If <data-path> contains any folders which do not fit the sequence or contains any files, an error occurs.

The sub-folders will be assumed to be individual job inputs and they will be uploaded as such in a round-robin fashion. For example, if <data-path> has three child folders and five jobs are being created the folders will be used as job input data in the following manner:

job 1 uses folder 1
job 2 uses folder 2
job 3 uses folder 3
job 4 uses folder 1
job 5 uses folder 2

When (-r|--recurse) and (-s|--sequence) are combined, the top-level directory must contain only folders which are named in sequence. Each sequence named sub-folder is then treated as an individual job input and read recursively (see above for how recursive directories are handled).

When <number-of-jobs> is a non-numeric value or a value less than 1, an error occurs and no jobs are generated.

When <data-path> is a file and is (-s|--sequence) provided, an error occurs and no jobs are generated.

When <number-of-jobs> is greater than 1, (-w|--delay) can be used to provide a <delay-seconds> value. <delay-seconds> is assumed to be zero, unless (-w|--delay) is provided. The <delay-seconds> value is used to force the generator to wait at least <delay-seconds> between sending jobs to <target>.

When <delay-seconds> is a non-numeric value or a value less than 0, an error occurs and no jobs are generated. target subcommand

cpload target <verb>

cpload target list prints the list known targets.

cpload target set [(-f|--force)] <name> [<url>] sets the current active remote target by <name>.

If <name> is unknown, then <url> is required and an error if not provided. When <url> is provided, it is associated with <name> and stored for future use. If a target named <name> exists, (-f|--force) must be provided or an error occurs.

The value of <name> must be at least a single character, no longer than 255 characters, cannot contain any white-space or control characters, does not start with a - character, and cannot contain any of the following characters: ! @ # $ % ^ & * ( ) < > = ? ' "

cpload target rm <name> removes the target named <name>. If no target named <name> exists an error occurs.