Monitoring: Job Accounting
12.1 Introduction
In addition to the concept of metrics for devices (Chapter 10), or the concept of metrics for jobs (Chapter 11), there is also the concept of metrics gathered for a classifier entity, for resources used during jobs. This last one is typically metrics gathered per user, for resources used during jobs.
Classifier-based metrics gathering for jobs can use classification done with PromQL queries on labeled entities (sections 12.2- 12.7), or it can be done with CMDaemon database queries (section 12.8).
Classifier-based metrics gathering for jobs is more conveniently called job accounting, partly because it ressembles the idea of an accountant watching over users to track their resource use while they carry out their jobs.
The concept, implementation, analysis, and visualization of job accounting are described in this Chapter.
For example, in BCM jobs resource usage can be presented per user. Thus, if there are jobs in a queue that are being processed, then the jobs can be listed:
Example
The resource usage statistics gathered per user, for example for a user tim, can then be analyzed and visualized using the job accounting interface of Base View (section 12.5).
12.2 Labeled Entities
In job accounting, job metrics during a run are tagged with extra labels, such as the job ID, host name, and the user running the job. The modified job metrics object that is tagged in this way then becomes a job accounting-related object, called a labeled entity.
Administrators interested in using job accounting can simply skip ahead and start reading about the Base View job accounting interface in 12.5, and just explore it directly. Those who would prefer some background on how job accounting is integrated with BCM and PromQL, can continue reading this section (12.2) and the next one ( 12.3).
12.2.1 Dataproducers For Labeled Entities
To view labeled entities in cmsh, the path to the labeledentity submode is:
cmsh > monitoring > labeledentity
The labeledentity submode allows job accounting-related objects, called labeled entities, to be viewed. The labels are in the form <key>="<value>”, for example: hostname="node001", or user="alice".
The default, existing labeled entities are created from the built-in JobSampler and JobMetadataSampler dataproducers when a job is run. Custom samplers, of type prometheus, can be used to create further custom labeled entities. A custom sampler dataproducer, for example customsamplerextras, can be created from the monitoring setup mode of cmsh as follows:
Example
The customsamplerextras dataproducer can now have its properties configured and committed as described in section 10.5.4.
12.2.2 PromQL And Labeled Entities
Labeled entities can be used by administrators to help create and debug job-related queries in the Prometheus query language, PromQL. PromQL is a part of the Prometheus monitoring and alerting toolkit (https://prometheus.io). Basic PromQL documentation is available at https://prometheus.io/docs/prometheus/latest/querying/basics/.
12.2.3 Job IDs And Labeled Entities
Each job ID has a number of labeled entities associated with it. Since the number of labeled entities scales with the number of nodes and jobs, the number of labeled entities can be very large. Therefore, if examining these entities using the CMDaemon front ends such as cmsh or Base View, then filtering or sorting the output is useful. For example, labeled entities associated with node001, and with the JobSampler data producer, and with job 1329 from the preceding output, could be viewed by filtering the full list of labeled entities as follows (output truncated and ellipsized):
Example
12.2.4 Measurables And Labeled Entities
The measurables (metrics) for an entity can be listed with the measurables (or metrics) command. For a particular entity with a JobSampler property and index value of 45447, the command can be run as follows:
In the labeledentity mode of cmsh, the measurables listing command, which lists the measurables for labeled entities, should not be confused with the measurable navigation command, which brings the administrator to the measurable submode under the main monitoring mode.
12.3 PromQL Queries
12.3.1 The Default PromQL Queries…
By default there are several predefined PromQL queries already available. The queries can be listed from the query submode:
Example
The queries can be conceptually divided into their classes, which at the time of writing of this section (January 2026) are: accounting, cluster, container, jobs, kubernetes, along with various drilldown levels which classifies the queries according to various groups. The grouping for drilldown levels can be confusing, and the drilldownoverview command (section 12.7.1) can be helpful in clarifying the query intention for these.
By default, queries of all classes are sampled over a period, except for the accounting and drilldown metrics.
A metric in the accounting class query is evaluated (interpolated) from existing values. These existing values are raw samples gathered over the period, up to the time when the query is evaluated.
12.3.2 …And A Short Description Of Them
The description of each query can be listed with a little cmsh and unix text utility juggling:
Example
This yields the following table:
The listings give an idea of what the query does.
For example, for the users_job_cpu_usage utility, the idea is that it shows the CPU usage for jobs for each user.
12.3.3 Modifying The Default PromQL Query Properties
The properties of a particular query can be shown and modified:
Example
The PromQL query code itself is typically a few lines long, and can also be viewed and modified using get, and set.
12.3.4 An Example PromQL Query, Properties, And Disassembly
The users_job_cpu_usage query is a standard predefined query, and is used as an example here. The query shows the CPU usage by a user around the time the sample was taken. It is sometimes called an “instantaneous” value. However it is not that instantaneous, because its value is calculated by taking samples of the CPU usage over the last 10 minutes of the job run rather than at the query time. The code for the query can be viewed with:
Example
For those unfamiliar with PromQL, some disassembly of the users_job_cpu_usage query is helpful.
Terminology used by PromQL and BCM, for the pieces used to build the query, is listed in the following table:
As was mentioned before: job account metrics, unlike traditional metrics, are not directly associated with the device-related objects. For such metrics, the monitoring data command dumpmonitoringdata is therefore not accessed in cmsh from device mode or category mode.
Instead, the Prometheus metric job_cpuacct_usage_seconds, for example, is accessed via the labeledentities mode.
The properties and interpolated values at a particular instant of time for the metric can be accessed via an instantquery such as (some output excised for clarity):
Example
The properties and interpolated values of the metric over a range of time can be accessed via a rangequery such as (some output excised for clarity):
Example
The association of the instantquery and rangequery output with job accounts is because its dataproducer is JobSampler.
Further options for the instantquery and rangequery commands can be found in their help texts within cmsh.
Time specifications for the instantquery and rangequery commands is covered further in section 12.3.7.
12.3.5 Aside: Getting Raw Values For A Prometheus Class Metric
The PromQL language is aimed at providing an overall view of jobs and resource usage. The actual individual raw values that Prometheus metrics are built on—the entries in the Time Series Database (TSDB)—are not regarded as being important for the end user. The emphasis in PromQL is on seeing the values as seen by statistical reworking.
This section, which is about the raw TSDB values, is thus provided as background information for administrators who would anyway like to see what the raw values look like.
Raw values of the metric for a job ID can be accessed by using the index of the labeled identity that is associated with that job ID. For example, job ID 624 can have its index found with some grepping (some output elided or excised for clarity):
Example
The index for the job ID 4 is da2b for node002. The job ID can be used by the dumpmonitoringdata command to show the series raw values along with their time stamps:
Example
These raw values are the values that are used for interpolation during PromQL queries.
The label names for job samples can be seen using the index:
12.3.6 …An Example PromQL Query, Properties, And Disassembly (Continued)
Getting back from the aside about raw values, and continuing on with the example PromQL query from the start of this section (here), the query code for the query users_job_cpu_usage was:
With the necessary background explanations having been carried out, the disassembly of this query can now be done:
The core of the query is built around the job sampler metric job_cpuacct_usage_seconds.
The irate measurement in this case calculates the rate of change based on the last two most recent values in the Prometheus range vector.
The Prometheus range vector is formed from the Prometheus instant query by using the square brackets with a time value enclosed (job_cpuacct_usage_seconds[10m]). The Prometheus instant query is, as the terminology table earlier pointed out, the Prometheus version of the job sampler metric.
Getting back to the range vector, a range vector in general is a series of values formed from the corresponding Prometheus instant query. With the instant query being job_cpuacct_usage_seconds here, the range vector is formed over a span of 10 minutes.
After the irate function has taken the average, the resultant is what in PromQL is called an instant vector value. This consists of data in the form {CPU seconds consumed during period, timestamp associated with time period sample}. The instant vector value is then joined against each vector element of the pair job_id, hostname to generate the labeled identifier for the job running on the node. The group_right of the result uses the wlm, job_id and hostname as the leading labels in the label identifiers. The job_metadata_is_running function means that values are generated only while job_metadata is running. The sum by(user) function means that the metric is aggregated over all raw data and grouped by user.
Visualisation based on the result is most easily carried out by plotting job CPU usage for each user against time in the period specified, which can be done in a more user-friendly way with Base View.
The users_job_wall_clock_seconds query, is similar, and can be used to plot wall clock seconds consumed by a user over the last period:
Example
Predefined queries can be executed in the labeledentity mode with the -q option:
Tab-completion for the -q option reveals the possible Prometheus queries. Regular measurables can be viewed without the -q option.
If no period is specified with the -p|--parameter option, then the default period is 1w as the output indicates.
If Base View is used instead of cmsh, then Prometheus queries can be selected via the navigation path:
Menu bar > Accounting and reporting icon > Monitoring > PromQL Queries > Show/Hide query selection
For example the users_job_wall_clock_seconds query can be selected, its query parameter can be set to 1 week, the change saved, and the query run.
12.3.7 Time Specification With PromQL
PromQL allows a variety of time specifications.
In Base View there is a monitoring interface dashboard display that is independent of the cmsh display (section 12.5), and times can be specified for range and instant queries there.
In cmsh, time specifications that can be set for the rangequery and instantquery queries can use some additional time specifications, as shown in the following examples:
Example
*The meaning can be understood by looking at how /d operates. It rounds off the day by truncation, which in the case of now-0d/d means the midnight prior to now.
**Similarly, yesterday’s midnight is the midnight immediately prior to 24 hours ago.
The time units for PromQL are (https://prometheus.io/docs/prometheus/latest/querying/basics/#range-vector-selectors):
-
s- seconds -
m- minutes -
h- hour -
d- day -
w- week -
M- month (31 days).Mis not a PromQL unit, so it cannot be used inside a query. But it is a handy alias in BCM for an invariant time of 31 days. -
y- year
The - operator in Prometheus is an offset operator, used only after now in the time specification field. The / <time unit> syntax implies a start at that unit of time.
Thus if, for example, now-1d/d is set as the end time, then when the query runs, it picks up the values at “yesterday’s midnight.” For each user, the value is the number of seconds the job of the user was in a wait state for the week prior to yesterday’s midnight. It rounds off the day by truncation, which in the case of now-1d/d means the midnight prior to 24 hours ago. Varying the user means that the number of seconds varies accordingly.
12.4 Parameterized PromQL Queries
It is also possible to create parameterized queries using the <key>="<value>” labels in the labeled entities mode.
This is handy for running the same query with different parameters or other drilldown options.
For example, an existing unparameterized query
users_job_wall_clock_seconds
can be used as a starting point:
For convenience, the original query can be cloned over to a new, soon-to-be-parameterized, query called
users_job_wall_clock_per_account_seconds
using
The idea is that Slurm accounts become a parameter in the new query.
Parameter fields can now be added to the query. All fields in users_job_wall_clock_per_account_seconds can be replaced verbatim. So any part of the query can be made into a parameter.
Some further adjustments are:
The new query can then be run by the administrator on demand.
The original query sums over all accounts:
The parameterized query lets the administrator run the same query for specific accounts.
If Slurm accounts for physics phys and mathematics math have been created with
then account=phys and account=math, are the <key>="<value>” format options. The query can then be run with the -p|--parameter option as follows:
12.4.1 Two Job GPU Metrics Used In PromQL Queries
There are two important job GPU metrics (section G.1.8.4) that are used in several PromQL queries. These two are:
-
The
job_gpu_utilizationGPU metric:This is based on the
gpu_utilizationmetric collected on GPU nodes via the DCGM library. The values it takes are in the interval [0, 1]. In DCGM the name of the metric isDCGM_FI_DEV_GPU_UTIL, and it represents the total GPU utilization.The
gpu_utilizationmetric values collected on the GPU are mapped to the job that uses the GPU at the time of the collection.job_gpu_utilizationis a labeled entity, tagged with a label representing a job. This allows the person carrying out PromQL queries to use such parameters asjob id,job user, and so on. It is assumed that one GPU is used only by processes of a single job, but there can be several GPUs, each used by different jobs simultaneously. Each of the GPUs on the node has independent values ofjob_gpu_utilization.One example of a PromQL query that uses the
job_gpu_utilizationGPU metric isjob_names_usage_gpu, which has the query expansion: -
The
job_gpu_wastedGPU metric:The
job_gpu_wastedmetric is a labeled entity (tagged metric) and is based on thejob_gpu_utilizationmetric. Thejob_gpu_wastedmetric shows what fraction, out of 1, of the GPUs on the node were unused by a job despite being allocated by the workload manager for that job. It can take values in the interval [0, 1].It is calculated as follows:
1 - (all_gpus_utilization / requested_gpus)
where
-
all_gpus_utilizationis the average utilization for allgpu_utilizationmetric values collected in the interval [-1; +1] from the time of the metric calculation for all GPUs requested by the job on the node -
requested_gpusis the number of GPUs allocated for the job in the workload manager
For example:
If a job requests two GPUs on a node, and the job does not use any of those GPUs at all, then
job_gpu_wastedtakes the value1.If the same job uses half of the first GPU and does not use the second one, then the metric value is calculated as:
1 - ((0.5 + 0) / 2) = 0.75
One example of a PromQL query that uses
job_gpu_wastedisjobs_wasted_allocated_gpus, which has the query expansion: -
12.5 Job Accounting In Base View
Job accounting in Base View is designed to present accounts of jobs without having to construct command lines with syntaxes that can be tricky to deal with. One useful output format is as basic Excel-format spreadsheets.
Job accounting can be viewed within Base View’s accounting mode available using the top right hand side of the Base View standard display (figure 10.5).
By default, job accounting opens up with a dashboard panel called K8S Container Metrics, which provides Kubernetes container metrics widgets (figure 12.1):
Figure 12.1: Job accounting: K8S container metrics panels
If Kubernetes containers have been running jobs that have been sampled, then pre-selected container-related queries can be run from these widgets, and the data samples can be displayed as tables or plots. If Kubernetes containers have not been running jobs, then the pre-selected container-related queries have no data samples to display (figure 12.1).
-
By default, the dashboard panel has some pre-selected queries related to user jobs over the last period.
-
Containers' Memory Usage in Bytes -
Containers' Total CPU Usage in Bytes -
Containers' Total Received Bytes -
Containers' Total Filesystem Usage in Bytes
The default layout requires scrolling down to view the other queries. Other layouts can be selected from the ·s
Actions > Edit Dashboardmenu option (figure 12.1).The dashboard can be managed generally using the same button, to also reach menu options to delete, clone, reset, export, and import dashboards.
Clicking the
+ Add Dashboardbutton next to theAccountingheader creates an entirely new dashboard panel within accounting. -
-
Clicking the
+Add Widgetbutton for a BCM dashboard panel, for example theK8S Container Metricspanel in figure 12.1, creates a new widget in the dashboard panel.- A query can be selected for the widget by right-clicking on the collapsed menu associated with it, using the navigation path : ⋮ >
(figure 12.2):
- A query can be selected for the widget by right-clicking on the collapsed menu associated with it, using the navigation path : ⋮ >
Figure 12.2: Job accounting: widget settings selected from ⋮ menu
A query can be either be a selected query, using the SELECT QUERY tab, or it can be a constructed query, using the PLAIN TEXT QUERY tab:
- For a selected query (figure 12.3), the
QueryandQuery previewbox are initially blank. A dropdown menu allows a query to be selected and saved. In figure 12.3 the selected query isgpu_power_usage_for_account:
Figure 12.3: Job accounting: query selection for widget
- For a constructed query (figure 12.4), the text can be constructed in the
Querybox and saved:
Figure 12.4: Job accounting: query construction for widget
12.5.1 Management And Use Of The Accounting Panel
The following description is illustrated by figure 12.1:
A Base View job accounting dashboard is made up of collections of Base View job accounting panels, also known as widgets.
-
To add a new dashboard, the
+ Add Dashboardbutton next to theAccountingheader can be clicked. Other dashboard management actions can be carried out with the·s{}`` Actionsmenu options. -
To add a new panel, the
+ Add Widgetbutton next to theAccountingheader can be clicked. Other widget management actions can be carried out with the⋮{}menu options.
Job accounting is intended to present accounts of jobs. So, when a new panel is created, the ⋮ > Widget Settings menu option (figure 12.2) displays a dialog that asks for inputs such as the range in which the report is to be carried out, the sampling interval, and the report name (figure 12.3).
If the dialog has SELECT QUERY set (figure 12.3), then PromQL queries can be selected from a dropdown menu in the Query box (figure 12.5):
Figure 12.5: Job accounting: query selection for widget
By default, when the new dashboard is saved, the selected queries are run and a table can be seen of the results.
The results of a run can be displayed according to the widget settings options of NOW, INSTANT, and RANGE (these options can be seen in figures 12.3 and 12.4).
- Selecting
NOWlets the output be displayed for the current time, as a pie chart (figure 12.6):
Figure 12.6: Job accounting: PromQL query pie chart display for the current time
or as a table (figure 12.7):
Figure 12.7: Job accounting: PromQL query table display for the current time
-
Selecting
INSTANTlets the output be displayed as a pie chart or as a table too, but for a specified time in the past. -
Selecting
RANGElets the output be displayed for a specified time range. This can be as a graph (figure 12.8)
Figure 12.8: Job accounting: PromQL query range graph display for a specified time range
or as a table (figure 12.9)
Figure 12.9: Job accounting: PromQL query range table display for a specified time range
For each option, the table display can be downloaded as a CSV file by clicking on Export which is convenient for plotting custom graphs in forms other than what Base View offers.
12.5.1.1 The Range Query Type
The range query is accessible in Base View by clicking on the RANGE option in figures 12.3 or 12.4. The time range over which the query is executed can then be set. On saving, a value is fetched for each interval within that time range, and plotted. Very long ranges can be computationally expensive.
The query submitter can set absolute Start time and End time parameters using the Base View navigation path (figure 12.10)
Accounting > ⋮ > Widget Settings > RANGE > Range > ABSOLUTE MODE
Figure 12.10: Job accounting: setting a range start time and end time
The query users_job_memory_usage_bytes is an example of the range query type. It fetches the total memory usage for jobs over intervals, grouped per user during that interval (figure 12.11).
Figure 12.11: Job Accounting: User Values Over A Range
12.5.1.2 The Interval Setting
The interval is the plot interval along the time axis. When editing a range query type, the value set in the interval field for a range query run is set when the Save button is clicked. Lower numbers lead to smaller intervals, hence higher resolution and higher query execution times. For interval in cmsh, a value of 0 means that a reasonable default interval is attempted. Setting 0 is not valid in Base View plots.
Choosing an interval that is less than the sampling time of the metric is a bit pointless, and tends to lead to data values that display non-smooth behaviour.
12.6 Access Control For Workload Accounting And Reporting
The ability to view jobs is controlled by four tokens defined in a user’s profile:
-
GET_JOB_TOKEN: Allows all running jobs to be seen. -
GET_OWN_JOB_TOKEN: Allows owned running jobs to be seen. -
GET_JOBINFO_TOKEN: Allows all cached historic and running jobs to be seen. -
GET_OWN_JOBINFO_TOKEN: Allows owned cached historic and running jobs to be seen.
To retrieve monitoring data for the job, the token PLOT_TOKEN must also be defined for the profile.
A job is always owned by the user that runs it. Ownership of a job can also be shared with other users by defining project managers. This establishes a 2-level hierarchy, with project managers above the subordinates, who are users that are assigned to the project manager. One or more accounts, can be assigned to project managers.
12.6.1 Defining Project Managers Using Internal User Management
Any user can be turned into a project manager. If the BCM LDAP server is being used, then the project manager can be configured via cmsh.
Example
alice can be made the project manager of bob and charlie. This allows her access to the job data of her subordinates:
Example
albert can be made the manager of the physics account. This gives him access to all jobs running under that account:
Both mechanisms, users and accounts, can be combined to provide access control.
Example
To limit access to jobs that are running under the physics account to a specific set of users:
12.6.2 Defining Project Managers Using External User Management
If an external user management server is used instead of the BCM LDAP server, then project managers cannot be defined in cmsh. Instead, a script has to be written that provides the definitions for project managers in the form of a JSON object.
The full path of the script, for example /path/to/the/script, has to be set as a value to the ProjectManagerScript parameter in cmd.conf. This is done by adding it to the AdvancedConfig directive (here):
Example
The directive becomes active after restarting CMDaemon.
An example of a project manager script can be found at /cm/local/apps/cmd/scripts/cm-project-managers.py. It gives users access to each other’s jobs if they share at least one group.
The easiest way to use the script is use a mapping to inform CMDaemon which of the other user’s jobs each user has access to.
Example
So if frank can access data belonging to bob and dennis, while bob can access data belonging to dennis, while dennis can only access his own data, then the project manager configurations can be set up as:
Account access control can also be included in the output of the script, by setting values for the users, accounts, and the boolean operator (and, or) options:
Example
After restarting CMDaemon, it automatically runs this script when committing a change for a device or data producer. It is also possible to manually trigger the script to be run on the active head, by executing:
The script must not take more than a few seconds to process.
12.6.2.1 Workaround For Project Manager Script That Takes Too Long
If the script takes longer to run, then it must be run outside of CMDaemon, and its output should be saved as a file. If the output file is located at /path/to/the/file, then its path can be set as an input to the ProjectManagerFile parameter in cmd.conf. This is done by adding it to the AdvancedConfig directive (here):
Example
The project manager definitions become active after CMDaemon is restarted.
12.7 Drilldown Queries For Workload Accounting And Reporting
Metrics can be classified in various ways. Common ways are by:
-
device: A typical hardware device is a node. Each node can then have its metrics, which are CPU usage, memory, storage, and other resource use, displayed over time. Device metrics are largely covered in this chapter in the sections up to and including section 10.8.
-
job: With this classifier, each job that is run by a workload manager can have its metrics, which are CPU usage, memory, storage and other resource use, displayed over time. Job metrics are covered in Chapter 11.
Other ways of classifying metrics are part of workload accounting. With workload accounting, a workload manager runs jobs, and a job metric can be classified by:
-
user
-
job (job ID)
-
account
-
job name
The classification can be carried out singly. However, it can also be carried out at the same time, like filters. For example:
-
each user could be classified for a particular job metric
or -
a particular user could be classified for a particular job metric
or -
a particular user could be classifed for a particular job metric for a particular job ID only
or -
a particular user could be classified for a particular job metric for that particular job ID only for a particular account only
A cluster administrator that uses several filters to get to the “bottom” of how resources are being used, functions in a manner reminiscent of someone drilling to the bottom to find something. This type of filtering is therefore called drilldown. Each filter corresponds to a level of drilldown.
Drilldown is a bit like how in cmsh the use of the filter command within jobs mode can narrow down what is displayed:
Example
except that in cmsh the value of the job metric is not specified or shown by the filter command.
Drilldown is also rather similar in concept to how pivot tables are used in Excel spreadsheets. Pivot tables are particular selections (like the filtered choices in drilldown). The selections are applied to a great deal of raw data. A function (like the metric in drilldown) is applied to the selection, to present the information more clearly to the end viewer.
In contrast with the filter output of cmsh, in Base View the job metric value is made visible in a table or in graphs over the period.
12.7.1 The drilldownoverview Command
The list of predefined drilldown queries of section 12.3.1 can be listed with the drilldownoverview command. This can be run from the query submode of the monitoring mode. The output, with some columns cut out for convenience, looks like:
Example
The queries have drilldown options. For example, the job_information_by_account:
12.8 The grid Command For Job Accounting
A way to carry out job accounting without relying on PromQL queries is the grid command of cmsh. The grid command can be accessed from within wlm mode for a workload manager. The command displays the nodes in a grid.
An example of grid output can be seen in the session here.
As seen in that output, the nodes are displayed in a sequence of rows. For each row, the following row of nodes has the same sequence of nodes, but after an interval of time. Each node in the grid is displayed as a colored block called a timeblock. The timeblock displays a value for the classifier entity that the grid command associates with that node. The value for the timeblock is indicated by either its color, or by text superimposed on that timeblock, or both.
12.8.1 The grid Command Help Text
The command options to grid can be looked up in the help text:
Example
The classifier entity shown by grid can be called its mode, and it can be set with the --mode option. By default the mode is set to used, which means the classifier entity value displayed for the node is either that it is being used to run a job, or not being used to run a job.
A help text to describe modes available for a timeblock can be seen with:
Example
12.8.2 Some grid Command Examples
With the grid command, the timeblock value is given a color, and can also be indicated by an associated text value.
12.8.2.1 Displaying Nodes Used
For example: over intervals of 10 minutes (600 seconds), from 28 hours ago to 27 hours ago, for nodes node001, node002, node003, the used mode for nodes can be displayed with the command:
Example
The --legend option provides a color legend after the display, and the --text option for this mode overwrites the timeblock with an associated value of 1.00000 or 0.000000.
The time specification is explained in detail in section 12.8.3.
12.8.2.2 Displaying Users Usage Of Nodes
Another example, which displays users that used the node the most, and their usage as a percentage, per timeblock of 600s, over a time from 5370 minutes in the past to 5350 minutes in the past:
The display percentage can be disabled with the --no-percent option.
12.8.2.3 Displaying Larger Numbers Of Nodes
For larger clusters, the administrator can view the grid output over a larger monitor, or over several monitors, and perhaps using higher resolutions to get a colour map. Figure 12.12 shows an output for 128 nodes:
Figure 12.12: Job Accounting: Grid account output for 128 nodes
The watch command of cmsh can be used to display updates as the cluster is used during the day, without requiring input from the cluster administrator. Patterns of use may be viewed in this manner.
For example, the black parts of node rows in figure 12.12 are a visual indication of times when those nodes were not used.
12.8.3 The grid Command Time Specification
The grid command time specification is identical to that of the dumpmonitoringdata command (section 10.6.4), except that dumpmonitoringdata has implicit mandatory arguments. The time specifications for grid are, on the other hand, done explicitly, and use the time options and arguments:
-
--end<end-time>and
-
--start<start-time>
The syntax for the time specifications of grid is also used by the statistics and filter commands.
The time options --start and --end can have their arguments specified as follows:
-
Fixed time format: The format for the times that make up the time pair can be:
-
[YY/MM/DD] HH:MM[:SS]
(If YY/MM/DD is used, then each time must be enclosed in double quotes) -
The unix epoch time (seconds since 00:00:00 1 January 1970)
-
-
now: For the--endoption, a value ofnowcan be set. The time at which thegridcommand is run is then used. -
Relative time format: One item in the time pair can be set to a fixed time format. The other item in the time pair can then have its time set relative to the fixed time item. The format for the non-fixed time item (the relative time item) can then be specified as follows:
-
For the <start-time>, a number prefixed with “-” is used. It indicates a time that much earlier than the fixed end time.
-
For the <end-time>, a number prefixed with “+” is used. It indicates a time that much later than the fixed start time.
-
The number values also have suffix values indicating the units of time, as seconds (
s), minutes (m), hours (h), or days (d).
The relative time format is summarized in the following table:
-
-
Both <start-time> and <end-time> can have their values prefixed with a “-”. In this case, the range over which the monitored values are seen is in the past, relative to the current time. If the end time for the range is specified as further in the past than the starting time, then the time values are swapped over so that the end time becomes more recent than the starting time.