Burning Nodes
The burn framework is a component of NVIDIA Base Command Manager 10 that can automatically run test scripts on specified nodes within a cluster. The framework is designed to stress test newly built machines and to detect components that may fail under load. Nodes undergoing a burn session with the default burn configuration, lose their filesystem and partition data for all attached drives, and revert to their software image on provisioning after a reboot.
11.1 Test Scripts Deployment
The framework requires power management to be running and working properly so that the node can be power cycled by the scripts used. In modern clusters power management is typically achieved by enabling a baseboard management controller such as IPMI, iLO, or Redfish. Details on power management are given in Chapter 4 of the Administrator Manual.
The framework can run any executable script. The default test scripts are mostly bash shell scripts and Perl scripts. Each test script has a directory in /cm/shared/apps/cmburn containing the script. The directory and test script must have the same name. For example: /cm/shared/apps/cmburn/disktest/disktest is the default script used for testing a disk. More on the contents of a test script is given in section 11.3.2.
11.2 Burn Configurations
11.2.1 Burn Configuration Simple XML Example
A burn configuration is an XML file stored in the CMDaemon database that specifies the burn tests and the order in which they run. Within the burn configuration the tests are normally grouped into sequences, and several sequences typically make up a phase. Phases in turn are grouped in either a pre-install section or post-install section. A simple example of such a burn configuration could therefore look like:
Example
11.2.1.1 Mail Tag
The optional <mail> tag pair can add a sequence of e-mail addresses, with each address enclosed in an <address> tag pair. These addresses receive burn failure and warning messages, as well as a notice when the burn run has completed.
11.2.1.2 Pre-install And Post-install
The pre-install part of a burn configuration is configured with the tag pair, and run from inside a node-installer environment. This environment is a limited Linux environment and allows some simpler tests to run before loading up the full Linux node environment.
Similarly, the post-install part of a burn configuration uses the tag pair to run from inside the full Linux node environment. This environment allows more complex tests to run.
11.2.1.3 Post-burn Install Mode
The optional tag pair allows the administrator to specify the install mode (section 5.4.4 of the Administrator Manual) after burn. The tag pair can enclose a setting of AUTO, FULL, MAIN, or NOSYNC. The default setting is the install mode that was set before burn started.
11.2.1.4 Phases
The phases sections must exist. If there is no content for the phases, the phases tags must still be in place (“must exist”). Each phase must have a unique name and must be written in the burn configuration file in alphanumerical order. By default, numbers are used as prefixes. The phases are executed in sequence.
11.2.1.5 Tests
Each phase consists of one or more test tags. The tests can optionally be passed arguments using the args property of the burn configuration file (section 11.2). If multiple arguments are required, they should be a space separated list, with the (single) list being the args property.
Tests in the same phase are run simultaneously.
Most tests test something and then end. For example, the disk test tests the performance of all drives and then quits.
Tests which are designed to end automatically are known as non-endless tests.
Tests designed to monitor continuously are known as endless tests. Endless tests are not really endless. They end once all the non-endless tests in the same phase are ended, thus bringing an end to the phase. Endless tests typically test for errors caused by the load induced by the non-endless tests. For example the mce_check test continuously keeps an eye out for Machine Check Exceptions while the non-endless tests in the same phase are run.
A special test is the final test, memtest86, which is part of the default burn run, as configured in the XML configuration default-destructive. It does run endlessly if left to run. To end it, the administrator can deal with its output at the node console or can power reset the node. It is usually convenient to remove memtest86 from the default XML configuration in larger clusters, and to rely on the HPL and memtester tests instead, for uncovering memory hardware errors.
11.2.2 Burn Configuration XML Schema Definition
In BCM, burn configuration setups such as in section 11.2.1 have their global structure defined using an XML schema definition, which is installed on the head node in /cm/local/apps/cmd/etc/htdocs/xsd/burnconfig.xsd.
The XML schema definition can also be viewed with:
Example
11.3 Running A Burn Configuration
Burn configurations can be viewed and executed from cmsh.
11.3.1 Burn Configuration And Execution In cmsh
11.3.1.1 Burn Configuration File Settings
From cmsh, the burn configurations can be accessed from partition mode as follows:
Example
The values of a particular burn configuration (default-destructive in the following example) can be viewed as follows:
Example
The set command can be used to modify existing values of the burn configuration, that is: Description, Name, and XML. XML is the burn configuration file itself. The get xml command can be used to view the file, while using set xml opens up the default text editor, thus allowing the burn configuration to be modified.
A new burn configuration can also be added with the add command. The new burn configuration can be created from scratch with the set command. However, an XML file can also be imported to the new burn configuration by specifying the full path of the XML file to be imported:
Example
The burn configuration can also be edited when carrying out burn execution with the burn command.
11.3.1.2 Executing A Burn
A burn as specified by the burn configuration file can be executed in cmsh using the burn command of device mode.
Burn commands: The burn commands can modify these properties, as well as execute other burn-related operations.
The burn commands are executed within device mode, and are:
-
burn start -
burn stop -
burn status -
burn log
The burn help text that follows lists the detailed options. Next, operations with the burn commands illustrate how the options may be used along with some features.
Burn command operations: Burn commands allow the following operations, and have the following features:
-
start,stop,status,log: The basic burn operations allow a burn to be started or stopped, and the status of a burn to be viewed and logged.-
The “
burn start” command always needs a configuration file name. In the following it isboxburn. The command also always needs to be given the nodes it operates on: -
The “
burn stop” command only needs to be given the nodes it operates on, for example: -
The “
burn status” command:-
may be given the nodes for which the status is to be found, for example:
-
-
The “
burn log” command displays the burn log for specified node groupings. Each node with a boot MAC address of <mac> has an associated burn log file.
-
-
Advanced options include the following:
-
-n|--nodes,-g|--group,-c|--category,-r|--rack,-h|--chassis,-e|--overlay,-t|--type,-i|--intersection,-u|--union,-r|--role,-s|--status: These options allow burn commands to be executed over various node groupings. -
--config<burn configuration>,--file<path>: The burn configuration file can be set as one of the XML burn configurations listed inpartitionmode, or it can be set as the path of the XML file from the file system -
-l|--later: This option disables the immediate power reset that occurs on running the “burn start” or “burn stop” command on a node. This allows the administrator to power down manually, when preferred. -
-e|--edit: The burn configuration file can be edited with the-eoption for the “burn start” command. This is an alternative to editing the burn configuration file inpartitionmode. -
--no-drain,--no-undrain: The node is not drained before burning, or not undrained after burning -
-p|--path: This shows the burn log path, by default (section 11.4) on the head node under a directory named after the mac address/var/spool/burn/<mac>.Example
-
Burn command output examples:
The burn status command has a compact one-line output per node:
Example
The fields in the preceding output example are:
Each test in a phase uses these letters to display its status:
The “burn log” command output looks like the following (some output elided):
The output of the burn log command is actually the messages file in the burn directory, for the node associated with a MAC-address directory <mac>. The burn directory is at /var/spool/burn/ and the messages file is thus located at:
/var/spool/burn/ <mac>/messages
The tests have their log files in their own directories under the MAC-address directory, using their phase name. For example, the pre-install section has a phase named 01-hwinfo. The output logs of this test are then stored under:
/var/spool/burn/ <mac>/01-hwinfo/
11.3.2 Writing A Test Script
This section describes a sample test script for use within the burn framework. The script is typically a shell or Perl script. The sample that follows is a Bash script, while the hpl script is an example in Perl.
Section 11.1 describes how to deploy the script.
11.3.2.1 Non-endless Tests
The following example test script is not a working test script, but can be used as a template for a non-endless test:
Example
11.3.2.2 Endless Tests
The following example test script is not a working test, but can be used as a template for an endless test.
Example
11.3.3 Burn Failures
Whenever the burn process fails, the output of the burn log command shows the phase that has failed and that the burn terminates.
Example
Here, burn-control, which is the parent of the disk testing process, keeps track of the tests that pass and fail. On failure of a test, burn-control terminates all tests.
The node that has failed then requires intervention from the administrator in order to change state. The node does not restart by default. The administrator should be aware that the state reported by the node to CMDaemon remains burning at this point, even though it is not actually doing anything.
To change the state, the burn must be stopped with the burn stop command in cmsh. If the node is restarted without explicitly stopping the burn, then it simply retries the phase at which it failed.
Under the burn log directory, the log of the particular test that failed for a particular node can sometimes suggest a reason for the failure. For retries, old logs are not overwritten, but moved to a directory with the same name, and a number appended indicating the try number. Thus:
Example
First try, and failing at 02-disks tests:
2nd try, after failing again:
3rd try, after failing again:
11.4 Relocating The Burn Logs
A burn run can append substantial amounts of log data to the default burn spool at /var/spool/burn. To avoid filling up the head node with such logs, they can be appended elsewhere.
11.4.1 Configuring The Relocation
The 3-part procedure that can be followed is:
- The
BurnSpoolDirsetting can be set in the CMDaemon configuration file on the head node, at/cm/local/apps/cmd/etc/cmd.conf. TheBurnSpoolDirsetting tells CMDaemon where to look for burn data when the burn status is requested throughcmsh.
CMDaemon should be restarted after the configuration has been set. This can be done with:
The burnSpoolHost setting, which matches the host, and burnSpoolPath setting, which matches the location, can be changed in the node-installer configuration file on the head node, at /cm/node-installer/scripts/node-installer.conf (for multiarch/multidistro configurations the path takes the form: /cm/node-installer-<distribution>-<architecture>/scripts/node-installer.conf). These have the following values by default:
These values define the NFS-mounted spool directory.
The burnSpoolHost value should be set to the new DNS host name, or to an IP address. The burnSpoolPath value should be set to the new path for the data.
- Part 3 of the procedure adds a new location to export the burn log. This is only relevant if the spool directory is being relocated within the head node. If the spool is on an external fileserver, the existing burn log export may as well be removed.
The new location can be added to the head node as a path value, from a writable filesystem export name. The writable filesystem export name can most easily be added using Base View, via the navigation path:
Devices > Head Nodes > Edit > Settings > Filesystem exports > Add
Adding a new name like this is recommended, instead of just modifying the path value in an existing Filesystem exports name. This is because changing things back if the configuration is done incorrectly is then easy. By default, the existing Filesystem exports for the burn directory has the name:
/var/spool/burn@internalnet
and has a path associated with it with a default value of:
/var/spool/burn
When the new name is set in Filesystem exports, the associated path value can be set in agreement with the values set earlier in parts 1 and 2.
If using cmsh instead of Base View, then the change can be carried out from within the fsexports submode. Section 3.13.1 of the Administrator Manual gives more detail on similar examples of how to add such filesystem exports.
11.4.2 Testing The Relocation
To test the changes, it is wise to first try a single node with a short burn configuration. This allows the administrator to check that install and post-install tests can access the spool directories. Otherwise there is a risk of waiting hours for the pre-install tests to complete, only to have the burn abort on the post-install tests. The following short burn configuration can be used:
Example
To burn a single node with this configuration, the following could be run from the device mode of cmsh:
Example
This makes an editor pop up containing the default burn configuration. The content can be replaced with the short burn configuration. Saving and quitting the editor causes the node to power cycle and start its burn.
The example burn configuration typically completes in less than 10 minutes or so, depending mostly on how fast the node can be provisioned. It runs the mprime test for about two minutes.