Monitoring: Job Chargeback
13.1 Introduction
13.1.1 The Word “Chargeback”
In a non-IT context, the term chargeback is commonly used with credit cards, in a situation where a cardholder disputes a payment that was made to a merchant. When the credit card company pays back the disputed payment to the card holder, that is called a chargeback.
In an IT context, the term chargeback still has to do with money, and the idea of getting money back. However, in practice the intention ideally is not about disputing costs, but rather about measuring what the costs are of the IT resources that have been requested. Measurement of requested resources means that there is a potential to charge back the users or groups of users who requested these resources.
So, for example, an IT department for an organization may be allocated a budget to run a cluster, meant for the benefit of the organization. The IT department may make the cluster available to many other departments. These other departments request cluster resources. The IT department measures the requested resources and charges back the associated costs.
If a cluster is used by several different departments in the organization, then a simple way to pay for resource requests is to spread the entire cost as a general overhead expense over all departments equally. That may make matters easy for the cluster administrator, but can be harmful to the organization, because without a fair resource request management, there is a tendency for resource request abuse.
If however the resource requests per department are measured, it means that department managers can be kept aware of how resource requests are being divided up. Being able to measure requested resources per department, and thus being able to charge back the department for the requested resources, means that the organization using the cluster can plan and manage resource request budgets efficiently and fairly.
13.1.2 Comparison Of Job Chargeback Monitoring Measurement With Other Monitoring Measurements
Monitoring measurements in BCM in general can be considered to be:
-
Monitoring of devices (Chapter 10), which is about monitoring devices in a cluster.
-
Job monitoring (Chapter 11), which is about using monitoring to measure the resources used by jobs that actually run.
-
Job accounting (Chapter 12), which is about using monitoring to measure the resources used by a user, a group, or other classifier entity.
-
Job chargeback (this chapter), which is about using monitoring to measure the resources requested for a job, whether the resources are used or not. This allows the requester to be charged the costs of requesting those resources.
The difference between actual resource use and requested resource use can be illustrated by a thought experiment:
First, a job is run that runs the CPU at 100% for 10 minutes. After that job is completed, a second job is run that runs a sleep command that lasts 10 minutes.
If these two jobs are considered from the point of view of CPU resource usage, then according to the workload manager:
The first job actually uses the CPU for 10 minutes, and the administrator can work out from the job monitoring system what user that ran that job, and charge the user for CPU usage.
For the second job, the administrator cannot use the job monitoring system to charge the user for the CPU usage because there was no significant usage measured. However, resources were used up during this time, because the request for the job blocked the availability of that CPU during this period for other users. The administrator would like to charge the user for preventing others from accessing the resources during that period. To do that requires measuring the resources requested during that period, rather than resources that were really used.
Thus, the aim of job chargeback monitoring is to provide a way to track resource requests, which typically differs a little from resource usage.
13.2 Job Chargeback Measurement
13.2.1 Predefined Job Chargebacks
Some predefined chargebacks can be listed and configured under the chargeback submode of the wlm mode.
The predefined list is:
13.2.1.1 Setting A Price For Resource Requests
The pricing can be set per resource requests over a specified time period. In the preceding list, the period is a month or a year. The resource and associated resource request consumption can be the following pairs:
Different workload managers use different resources for resource measurement, which is why there is a variety in the resources that can be used for pricing.
13.2.2 Setting A Custom Job Chargeback
In addition to the predefined job chargebacks, more chargebacks can be added. For example, the number of jobs completed so far today can be set up as follows:
Example
13.2.2.1 Chargeback Groupings
The grouping for the new chargeback Jobs completed this day grouped by user is set to be Group by user. Grouping by user is a common grouping, because finding resource use by an individual user is typically the most useful case. Grouping is possible by:
-
user
-
group
-
account
-
job name
-
job ID
-
accounting info
After setting up chargebacks to suit the needs of the cluster administrator, queries can be made and reports can be generated using chargebacks. The report and request commands (section 13.2.3) are used for this.
13.2.3 The report And request Commands
Continuing on with the chargeback jobs completed this day grouped by user created in section 13.2.2, the report and request commands can be used after CPU request data values have been gathered on the jobs being run.
13.2.3.1 The report Command And Its Options
The report command displays a table of the number of jobs that were run per grouping for a chargeback, alongside the resource use and cost for each grouping.
Thus, some time after running jobs in Slurm, the report output for the chargeback created earlier, jobs completed this day grouped by user, might look as follows (some columns elided for clarity):
Example
The help text for report command lists formatting options:
13.2.3.2 The request Command And Its Options
The request command lists the chargeback resources requested for a workload manager.
The request command can be run without options. In that case the output shows the resource request consumption for the chargeback, for the jobs over the period associated with that chargeback:
Example
The help text for request lists formatting, grouping, pricing, and filtering options:
For example, using epoch times to specify the start and end times, and grouping by user:
Example
Another way of duplicating the output of request without options, is to explicitly specify the default values. For the chargeback jobs completed this day grouped by user, which was set up in section 13.2.2. it corresponds to a start time of now/d and an end time of now/d:
Example
Users can be added to the table with the -u option (some output is truncated here for clarity in the examples that follow):
Example
A jobs drilldown can be carried out with -j.
Example
The jobs drilldown can be carried out for a particular user, alice, using the filter option:
Example
This can have the user fields added for the case when several users are specified, as follows:
Example
As the help text for request suggests, there are many more combinations possible.
13.3 Job Chargeback Background Information
Because users can run large numbers of jobs per day, the storage requirement for chargeback records can be very large indeed. For this reason, a cache is kept in memory, and flushed to storage in a MySQL database.
CMDaemon AdvancedConfig directives for configuring the MySQL storage for chargebacks are: