Hardware Recommendations

The hardware suggestions in section 3.1 are for a minimal cluster, and are inadequate for larger clusters.

For larger clusters, hardware suggestions and examples are given in this section.

The memory used depends significantly on CMDaemon, which is the main NVIDIA Base Command Manager service component, and on the number of processes running on the head node or regular node. The number of processes mostly depends on the number of metrics and health checks that are run.

Hard drive storage mostly depends on the number of metrics and health checks that are managed by CMDaemon.

B.1 Heuristics For Requirements

Normal system processes run on the head and regular node if the cluster manager is not running, and take up their own RAM and drive space.

B.1.1 Heuristics For Requirements For A Regular Node

A calculation of typical regular node requirements can be made as follows:

B.1.1.1 Regular Node Disk Size

For disked nodes, a disk size of around 25 GB is the minimum needed. 256GB should always be fine at the time of writing of this section (October 2023). The disk size should be large enough to hold the entire regular node image that the head node supplies to it, which typically is around 6GB, along with swap, log files and other local overhead for the jobs that will run on the regular node.

B.1.1.2 Regular Node Memory Size

The total RAM required is roughly the sum of:
RAMusedfornon-BCMsystemprocesses + 50MB + (numberof~nodes × 10kB).

B.1.2 Heuristics For Requirements For A Head Node

A calculation of typical head node requirements can be made as follows:

B.1.2.1 Head Node Disk Size

The disk size required is roughly the sum of:
spaceneededbyoperatingsystemwithoutclustermanager + 5GBperregularnodeimage + (100kB × numberofmetricsandhealthchecks × numberofdevices)

A device means any item seen as a device by CMDaemon. A list of devices can be seen by cmsh under its device mode. Examples of devices are: regular nodes, cloud nodes, switches, head nodes, GPU units, and PDUs.

B.1.2.2 Head Node Memory Size

The total RAM required is roughly the sum of:
RAMusedfornormalsystemprocess + 100MB + (numberof~nodes × 1.8MB)

This assumes less than 100 metrics and health checks are being measured, which is a default for systems that are just head nodes and regular nodes. Beyond the first 100 metrics and health checks, each further 100 extra take about 1MB extra per device.

B.2 Observed Head Node Resources Use, And Suggested Specification

B.2.1 Observed Head Node Example CMDaemon And MySQL Resources Use

CMDaemon and MySQL have the following approximate default resource usage on the head node as the number of nodes increases:

B.2.2 Suggested Head Node Specification For Clusters Beyond 1000 Nodes

For clusters with more than 1000 nodes, a head node is recommended with at least the following specifications:

• 24 cores

• 128 GB RAM

• 512 GB SSD

The extra RAM is useful for caching the filesystem, so scrimping on it makes little sense.

Handy for speedy retrievals is to place the monitoring data files, which are by default located under /var/spool/cmd/monitoring/, on an SSD.

A dedicated /var or /var/lib/mysql partition for clusters with greater than 2500 nodes is also a good idea.