> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Monitoring: Job Chargeback ## 13.1 Introduction ### 13.1.1 The Word “Chargeback” In a non-IT context, the term chargeback is commonly used with credit cards, in a situation where a cardholder disputes a payment that was made to a merchant. When the credit card company pays back the disputed payment to the card holder, that is called a chargeback. In an IT context, the term chargeback still has to do with money, and the idea of getting money back. However, in practice the intention ideally is not about disputing costs, but rather about measuring what the costs are of the IT resources that have been requested. Measurement of requested resources means that there is a potential to charge back the users or groups of users who requested these resources. So, for example, an IT department for an organization may be allocated a budget to run a cluster, meant for the benefit of the organization. The IT department may make the cluster available to many other departments. These other departments request cluster resources. The IT department measures the requested resources and charges back the associated costs. If a cluster is used by several different departments in the organization, then a simple way to pay for resource requests is to spread the entire cost as a general overhead expense over all departments equally. That may make matters easy for the cluster administrator, but can be harmful to the organization, because without a fair resource request management, there is a tendency for resource request abuse. If however the resource requests per department are measured, it means that department managers can be kept aware of how resource requests are being divided up. Being able to measure requested resources per department, and thus being able to charge back the department for the requested resources, means that the organization using the cluster can plan and manage resource request budgets efficiently and fairly. ### 13.1.2 Comparison Of Job Chargeback Monitoring Measurement With Other Monitoring Measurements Monitoring measurements in BCM in general can be considered to be: * Monitoring of devices (Chapter [10](/admin/monitoring)), which is about monitoring devices in a cluster. * Job monitoring (Chapter [11](/admin/monitoringjob)), which is about using monitoring to measure the resources used by jobs that actually run. * Job accounting (Chapter [12](/admin/monitoringaccounting)), which is about using monitoring to measure the resources used by a user, a group, or other classifier entity. * Job chargeback (this chapter), which is about using monitoring to measure the resources requested for a job, whether the resources are used or not. This allows the requester to be charged the costs of requesting those resources. The difference between actual resource use and requested resource use can be illustrated by a thought experiment: First, a job is run that runs the CPU at 100% for 10 minutes. After that job is completed, a second job is run that runs a `sleep` command that lasts 10 minutes. If these two jobs are considered from the point of view of CPU resource usage, then according to the workload manager: The first job actually uses the CPU for 10 minutes, and the administrator can work out from the job monitoring system what user that ran that job, and charge the user for CPU usage. For the second job, the administrator cannot use the job monitoring system to charge the user for the CPU usage because there was no significant usage measured. However, resources were used up during this time, because the request for the job blocked the availability of that CPU during this period for other users. The administrator would like to charge the user for preventing others from accessing the resources during that period. To do that requires measuring the resources requested during that period, rather than resources that were really used. Thus, the aim of job chargeback monitoring is to provide a way to track resource requests, which typically differs a little from resource usage. ## 13.2 Job Chargeback Measurement ### 13.2.1 Predefined Job Chargebacks Some predefined chargebacks can be listed and configured under the `chargeback` submode of the `wlm` mode. The predefined list is: ```bash [basecm11->wlm[slurm]]% chargeback [basecm11->wlm[slurm]->chargeback]% list Group by Group by Group by Price per Price per Name (key) user group account CPU second GPU second ------------------------------------------------------ -------- -------- -------- ---------- ---------- Jobs completed last month grouped by account no no yes 8.64 $/d 1.80 $/h Jobs completed last month grouped by group no yes no 8.64 $/d 1.80 $/h Jobs completed last month grouped by user yes no no 8.64 $/d 1.80 $/h Jobs completed last month grouped by user and account yes no yes 8.64 $/d 1.80 $/h Jobs completed this month grouped by account no no yes 8.64 $/d 1.80 $/h Jobs completed this month grouped by group no yes no 8.64 $/d 1.80 $/h Jobs completed this month grouped by user yes no no 8.64 $/d 1.80 $/h Jobs completed this month grouped by user and account yes no yes 8.64 $/d 1.80 $/h Jobs completed this year grouped by account no no yes 8.64 $/d 1.80 $/h Jobs completed this year grouped by group no yes no 8.64 $/d 1.80 $/h Jobs completed this year grouped by user yes no no 8.64 $/d 1.80 $/h Jobs completed this year grouped by user and account yes no yes 8.64 $/d 1.80 $/h ``` #### 13.2.1.1 Setting A Price For Resource Requests The pricing can be set per resource requests over a specified time period. In the preceding list, the period is a month or a year. The resource and associated resource request consumption can be the following pairs:
**Resource And Resource Request Consumption Pairs**
**Resource** **Resource Request Consumption**
CPU CPU second
GPU GPU second
CPU core CPU core second
slot slot second
memory bytes byte-second
Power Watt-second (Joules)
Different workload managers use different resources for resource measurement, which is why there is a variety in the resources that can be used for pricing. ### 13.2.2 Setting A Custom Job Chargeback In addition to the predefined job chargebacks, more chargebacks can be added. For example, the number of jobs completed so far today can be set up as follows: **Example** ```bash [basecm11->wlm[slurm]->chargeback]% add "Jobs completed this day grouped by user" [basecm11->...ck*[Jobs completed this day grouped by user*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name jobs completed this day grouped by user Revision Notes <0B> Group by user no Group by group no Group by account no Group by job name no Group by job ID no Group by accounting info Users Groups Accounts Job names Job IDs Accounting info < not set > Price per CPU second 0.00 $/s Price per CPU core second 0.00 $/s Price per GPU second 0.00 $/s Price per memory byte-second 0$/B*s Price per slot second 0$/slot*s Price per Watt second 0$/W*s Currency $ Start time now/M End time now/M UTC no Include running no Calculate prediction no Preference 0 [basecm11->...ck*[Jobs completed this day grouped by user*]]% set pricepercpusecond 0.0001$/s [basecm11->...ck*[Jobs completed this day grouped by user*]]% set groupbyuser yes [basecm11->...ck*[Jobs completed this day grouped by user*]]% set starttime now/d [basecm11->...ck*[Jobs completed this day grouped by user*]]% set endtime now/d [basecm11->...ck*[Jobs completed this day grouped by user*]]% commit ``` #### 13.2.2.1 Chargeback Groupings The grouping for the new chargeback `Jobs completed this day grouped by user` is set to be `Group by user`. Grouping by user is a common grouping, because finding resource use by an individual user is typically the most useful case. Grouping is possible by: * user * group * account * job name * job ID * accounting info After setting up chargebacks to suit the needs of the cluster administrator, queries can be made and reports can be generated using chargebacks. The `report` and `request` commands (section [13.2.3](#1323-the-report-and-request-commands)) are used for this. ### 13.2.3 The `report` And `request` Commands Continuing on with the chargeback `jobs completed this day grouped by user` created in section [13.2.2](#1322-setting-a-custom-job-chargeback), the `report` and `request` commands can be used after CPU request data values have been gathered on the jobs being run. #### 13.2.3.1 The `report` Command And Its Options The `report` command displays a table of the number of jobs that were run per grouping for a chargeback, alongside the resource use and cost for each grouping. Thus, some time after running jobs in Slurm, the `report` output for the chargeback created earlier, `jobs completed this day grouped by user`, might look as follows (some columns elided for clarity): **Example** ```bash [basecm11->wlm[slurm]->chargeback[jobs completed this day grouped by user]]% report # Start - Tue Sep 8 00:00:00 2020 CEST (1599516000) # End - Tue Sep 8 23:59:59 2020 CEST (1599602399) User Jobs Runtime (s) CPU (s) CPU ($) ... Price ($) ------------ ------------ ------------ ------------ ---------- ... ---------- alice 17 7,806 7,806 0.78 ... 0.78 bob 20 8,775 8,775 0.88 ... 0.88 charlie 19 7,007 7,007 0.7 ... 0.7 david 10 5,122 5,122 0.51 ... 0.51 edgar 25 10,502 10,502 1.05 ... 1.05 frank 21 8,289 8,289 0.83 ... 0.83 ``` The help text for `report` command lists formatting options: ```bash [basecm11->wlm[slurm]->chargeback]% help report Name: report - Create charge back report Usage: report [options] Options: -d, --delimiter Set default row separator -v, --verbose Be more verbose: multiline table --start Pagination start offset --limit Pagination result limit ``` #### 13.2.3.2 The `request` Command And Its Options The `request` command lists the chargeback resources requested for a workload manager. The `request` command can be run without options. In that case the output shows the resource request consumption for the chargeback, for the jobs over the period associated with that chargeback: **Example** ```bash [basecm11->wlm[slurm]->chargeback[jobs completed this day grouped by user]]% request # Start - Sun Sep 13 00:00:00 2020 CEST (1599948000) # End - Sun Sep 13 23:59:59 2020 CEST (1600034399) Jobs Runtime (s) CPU (s) Core (s) GPU (s) Slots (s) Memory (B*s) ------------ ------------ ------------ ------------ ------------ ------------ ------------ 5 1,765 1,765 0 0 0 0 ``` The help text for `request` lists formatting, grouping, pricing, and filtering options: ```bash [basecm11->wlm[slurm]->chargeback]% help request Name: request - Chargeback report for workload Usage: request [options] Options: -u, --group-by-user group by username --filter-user [,,...] filter on specified users -g, --group-by-group group by group name --filter-group [,,...] filter on specified groups -a, --group-by-account group by account --filter-account [,,...] filter on specified accounts -j, --group-by-job-name group by job name --filter-job-name [,,...] filter on specified job names -i, --group-by-job-id group by job id --filter-job-id [,,...] filter on specified job ids --group-by-accounting-info [,,...] group by specified accounting info fields --filter-accounting-info [,,...] filter on specified parent-ids --price-per-cpu-second Price per CPU second --price-per-cpu-core-second Price per CPU core second --price-per-gpu-second Price per GPU second --price-per-gpu-second Price per GPU second --price-per-memory-byte-second Price per memory byte * second --price-per-slot-second Price per slot second --currency Change the currency in which the price is displayed (default $) --include-running include running jobs in charge back report (prices will not be final) --calculate-prediction calculate a prediction for an incomplete time frame -d, --delimiter Set default row separator --sort [,,...] Override default sort order --start-time, -s